2024-07-26T23:15:08.027 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T23:15:08.046 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T23:15:08.156 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516 branch: wip-yuri-testing-2024-07-26-0628 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7820516' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri-testing-2024-07-26-0628 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 160 osd op thread timeout: 150 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2024-07-26-0628 sha1: 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b owner: scheduled_yuriw@teuthology priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 7667 sha1: 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri-testing-2024-07-26-0628 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b targets: smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGh/OTKb9BauDrSJETdT5PDHA6QPNmwZ98jZ+aFieEWYLPX9+XzaVkG2ryVoOCIo8mbA0AaN+BaE6/DnOKL+UQA= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHJs+7Jlsb2eLt1qygUU+szB+erFTL0fwmvrqchqqzrptQDDElYXbPgt/hjVBrw/JSwqprjkotVIpANJtZE9ys8= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_21:23:43 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T23:15:08.156 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa; will attempt to use it 2024-07-26T23:15:08.157 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa/tasks 2024-07-26T23:15:08.157 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T23:15:08.159 INFO:teuthology.task.internal:Checking packages... 2024-07-26T23:15:08.180 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b' 2024-07-26T23:15:08.180 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T23:15:08.180 INFO:teuthology.packaging:ref: None 2024-07-26T23:15:08.180 INFO:teuthology.packaging:tag: None 2024-07-26T23:15:08.180 INFO:teuthology.packaging:branch: wip-yuri-testing-2024-07-26-0628 2024-07-26T23:15:08.180 INFO:teuthology.packaging:sha1: 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b 2024-07-26T23:15:08.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri-testing-2024-07-26-0628 2024-07-26T23:15:08.319 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5372.g67f28c2a 2024-07-26T23:15:08.321 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T23:15:08.328 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T23:15:08.328 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T23:15:08.354 INFO:teuthology.task.internal:Saving configuration 2024-07-26T23:15:08.369 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T23:15:08.380 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T23:15:08.402 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 23:10:11.845786', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGh/OTKb9BauDrSJETdT5PDHA6QPNmwZ98jZ+aFieEWYLPX9+XzaVkG2ryVoOCIo8mbA0AaN+BaE6/DnOKL+UQA='} 2024-07-26T23:15:08.425 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 23:10:11.843719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHJs+7Jlsb2eLt1qygUU+szB+erFTL0fwmvrqchqqzrptQDDElYXbPgt/hjVBrw/JSwqprjkotVIpANJtZE9ys8='} 2024-07-26T23:15:08.426 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T23:15:08.431 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-07-26T23:15:08.432 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-07-26T23:15:08.432 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T23:15:08.514 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efdb89afbe0>, signals=[15]) 2024-07-26T23:15:08.514 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T23:15:08.521 INFO:teuthology.task.internal:Opening connections... 2024-07-26T23:15:08.521 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-07-26T23:15:08.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:15:08.598 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-07-26T23:15:08.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:15:08.675 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T23:15:08.682 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-07-26T23:15:08.700 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-07-26T23:15:08.700 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-07-26T23:15:08.756 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2024-07-26T23:15:08.756 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="9" 2024-07-26T23:15:08.756 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2024-07-26T23:15:08.756 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2024-07-26T23:15:08.756 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="9" 2024-07-26T23:15:08.756 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el9" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:LOGO="fedora-logo-icon" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T23:15:08.757 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T23:15:08.758 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-07-26T23:15:08.779 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-07-26T23:15:08.797 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-07-26T23:15:08.797 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-07-26T23:15:08.854 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-07-26T23:15:08.854 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-07-26T23:15:08.854 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-07-26T23:15:08.854 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T23:15:08.855 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T23:15:08.856 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-07-26T23:15:08.879 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T23:15:08.887 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T23:15:08.912 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T23:15:08.913 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-07-26T23:15:08.915 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-07-26T23:15:08.930 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T23:15:08.936 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T23:15:08.936 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-07-26T23:15:08.971 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-07-26T23:15:09.275 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T23:15:09.328 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T23:15:09.346 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-07-26T23:15:09.346 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-07-26T23:15:09.346 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T23:15:09.347 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T23:15:09.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T23:15:09.363 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-07-26T23:15:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T23:15:09.364 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-07-26T23:15:09.421 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-437.el9.x86_64 2024-07-26T23:15:09.421 INFO:teuthology.task.kernel:Running kernel on smithi139: 5.14.0-437.el9.x86_64 2024-07-26T23:15:09.421 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2024-07-26T23:15:09.422 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-437.el9.x86_64 2024-07-26T23:15:09.423 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-437.el9.x86_64 2024-07-26T23:15:09.423 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-07-26T23:15:11.916 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.2 MB 00:00 2024-07-26T23:15:13.004 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.2 MB 00:02 2024-07-26T23:15:14.844 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-07-26T23:15:19.633 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.5 MB 00:00 2024-07-26T23:15:20.329 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 20 MB 00:06 2024-07-26T23:15:21.276 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 18 kB 00:00 2024-07-26T23:15:22.188 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 28 MB/s | 22 MB 00:00 2024-07-26T23:15:27.112 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.5 MB 00:02 2024-07-26T23:15:27.519 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-07-26T23:15:28.613 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 18 kB 00:00 2024-07-26T23:15:29.550 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-07-26T23:15:29.551 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-07-26T23:15:29.551 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-07-26T23:15:29.656 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-07-26T23:15:29.666 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-07-26T23:15:29.667 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:15:29.667 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:15:29.667 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:15:29.667 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2024-07-26T23:15:29.667 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:15:29.667 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 84 M 2024-07-26T23:15:29.668 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 126 M 2024-07-26T23:15:29.668 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:15:30.028 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 6.2 MB/s | 1.5 MB 00:00 2024-07-26T23:15:30.304 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 33 MB/s | 17 MB 00:00 2024-07-26T23:15:30.687 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 45 MB/s | 30 MB 00:00 2024-07-26T23:15:31.004 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 29 MB/s | 36 MB 00:01 2024-07-26T23:15:31.004 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:15:31.004 INFO:teuthology.orchestra.run.smithi139.stdout:Total 63 MB/s | 84 MB 00:01 2024-07-26T23:15:31.739 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:15:32.087 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:15:32.087 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:15:33.058 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:15:33.059 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:15:34.833 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-07-26T23:15:35.498 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:15:36.159 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-07-26T23:15:36.245 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T23:15:36.822 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-07-26T23:15:36.823 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-07-26T23:15:36.906 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:15:36.913 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-07-26T23:15:36.914 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:15:36.915 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:15:36.915 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:15:36.915 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-07-26T23:15:36.915 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:15:36.915 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 84 M 2024-07-26T23:15:36.915 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 126 M 2024-07-26T23:15:36.915 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:15:37.859 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.8 MB/s | 1.5 MB 00:00 2024-07-26T23:15:38.467 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2024-07-26T23:15:38.915 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T23:15:39.053 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T23:15:39.184 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 23 MB/s | 30 MB 00:01 2024-07-26T23:15:39.510 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 14 MB/s | 36 MB 00:02 2024-07-26T23:15:39.512 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:15:39.513 INFO:teuthology.orchestra.run.smithi177.stdout:Total 32 MB/s | 84 MB 00:02 2024-07-26T23:15:40.249 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:15:40.604 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:15:40.604 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:15:41.627 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:15:41.627 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:15:43.915 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T23:15:44.045 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:15:44.088 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:15:44.751 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-07-26T23:15:44.852 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T23:15:47.518 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T23:15:47.666 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T23:15:48.903 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:15:52.507 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T23:15:52.642 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:15:57.504 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:18.950 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:18.991 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:19.310 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:19.310 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-07-26T23:16:19.310 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T23:16:19.310 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T23:16:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-5.14.0-480.el9.x86_64 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:16:20.151 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:16:20.560 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2024-07-26T23:16:20.652 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-26T23:16:20.653 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-26T23:16:20.653 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-07-26T23:16:20.653 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-07-26T23:16:20.654 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-07-26T23:16:20.654 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-07-26T23:16:20.654 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2024-07-26T23:16:20.654 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2024-07-26T23:16:20.710 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-26T23:16:20.711 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-26T23:16:20.711 DEBUG:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2024-07-26T23:16:20.741 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2024-07-26T23:16:20.741 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.06 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 68.el9 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Group : Unspecified 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 8264254 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : builder@centos.org 2024-07-26T23:16:20.742 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:16:20.743 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T23:16:20.744 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T23:16:20.744 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 5.14.0-480.el9.x86_64 2024-07-26T23:16:20.744 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T23:16:21.413 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2024-07-26T23:16:22.797 INFO:teuthology.orchestra.run.smithi139.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T23:16:22.823 INFO:teuthology.orchestra.run.smithi139.stderr:done 2024-07-26T23:16:22.825 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-07-26T23:16:22.841 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.ShDwAdCg7M 2024-07-26T23:16:22.841 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ShDwAdCg7M 2024-07-26T23:16:22.907 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.ShDwAdCg7M 2024-07-26T23:16:23.049 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.ShDwAdCg7M is 6KB 2024-07-26T23:16:23.097 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.ShDwAdCg7M 2024-07-26T23:16:23.111 DEBUG:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T23:16:23.181 INFO:teuthology.orchestra.run.smithi139.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T23:16:23.181 INFO:teuthology.orchestra.run.smithi139.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T23:16:23.181 INFO:teuthology.orchestra.run.smithi139.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T23:16:23.181 INFO:teuthology.orchestra.run.smithi139.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-07-26T23:16:23.182 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-07-26T23:16:23.357 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2024-07-26T23:16:27.207 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:27.247 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:27.624 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:27.624 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-07-26T23:16:27.624 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T23:16:27.624 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-480.el9.x86_64 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:16:28.435 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:16:28.806 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-07-26T23:16:28.895 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-26T23:16:28.896 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-26T23:16:28.896 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-07-26T23:16:28.896 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-07-26T23:16:28.896 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-07-26T23:16:28.896 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-07-26T23:16:28.896 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-07-26T23:16:28.896 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-07-26T23:16:28.954 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-26T23:16:28.955 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-26T23:16:28.955 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-07-26T23:16:28.986 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-07-26T23:16:28.986 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-07-26T23:16:28.986 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-07-26T23:16:28.986 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-07-26T23:16:28.986 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-07-26T23:16:28.987 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:16:28.988 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T23:16:28.989 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T23:16:28.989 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-480.el9.x86_64 2024-07-26T23:16:28.990 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T23:16:29.633 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-07-26T23:16:31.067 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T23:16:31.118 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-07-26T23:16:31.120 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-07-26T23:16:31.135 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.OYDSWYwVNk 2024-07-26T23:16:31.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OYDSWYwVNk 2024-07-26T23:16:31.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.OYDSWYwVNk 2024-07-26T23:16:31.340 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.OYDSWYwVNk is 6KB 2024-07-26T23:16:31.389 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.OYDSWYwVNk 2024-07-26T23:16:31.403 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T23:16:31.464 INFO:teuthology.orchestra.run.smithi177.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T23:16:31.464 INFO:teuthology.orchestra.run.smithi177.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T23:16:31.465 INFO:teuthology.orchestra.run.smithi177.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T23:16:31.465 INFO:teuthology.orchestra.run.smithi177.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-07-26T23:16:31.466 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-07-26T23:16:31.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-07-26T23:16:53.385 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-07-26T23:16:53.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-07-26T23:16:53.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:17:01.614 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-07-26T23:17:01.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-07-26T23:17:01.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:17:11.775 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-07-26T23:17:19.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-07-26T23:17:20.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-07-26T23:17:20.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:17:23.842 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-07-26T23:17:28.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-07-26T23:17:28.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:17:32.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-07-26T23:17:35.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-07-26T23:17:35.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:17:44.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-07-26T23:17:44.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:17:54.271 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-07-26T23:18:09.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-07-26T23:18:09.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:18:09.654 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-07-26T23:18:10.023 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-07-26T23:18:10.023 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-07-26T23:18:10.024 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-07-26T23:18:10.039 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-480.el9.x86_64 2024-07-26T23:18:10.039 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-07-26T23:18:10.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T23:18:10.039 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T23:18:11.040 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-07-26T23:18:11.040 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-07-26T23:18:11.128 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2024-07-26T23:18:11.163 DEBUG:teuthology.parallel:result is None 2024-07-26T23:18:15.916 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-07-26T23:18:16.318 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-07-26T23:18:16.318 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-480.el9.x86_64"... 2024-07-26T23:18:16.319 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-07-26T23:18:16.333 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-480.el9.x86_64 2024-07-26T23:18:16.333 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-07-26T23:18:16.333 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T23:18:16.333 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T23:18:17.334 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-07-26T23:18:17.334 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-07-26T23:18:17.409 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-07-26T23:18:17.443 DEBUG:teuthology.parallel:result is None 2024-07-26T23:18:17.443 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T23:18:17.453 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T23:18:17.453 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T23:18:17.456 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T23:18:17.473 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T23:18:17.484 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T23:18:17.534 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T23:18:17.534 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T23:18:17.537 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T23:18:17.693 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T23:18:17.700 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T23:18:17.700 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T23:18:17.702 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T23:18:17.731 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T23:18:17.742 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T23:18:17.743 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T23:18:17.754 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T23:18:17.755 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T23:18:17.764 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T23:18:17.764 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T23:18:17.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T23:18:17.821 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T23:18:17.849 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T23:18:17.849 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T23:18:17.856 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T23:18:17.877 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T23:18:17.958 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T23:18:18.015 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T23:18:18.097 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T23:18:18.141 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:18:18.141 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T23:18:18.207 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T23:18:18.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T23:18:18.309 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T23:18:18.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T23:18:18.445 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:18:18.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T23:18:18.515 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-07-26T23:18:18.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-07-26T23:18:18.561 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T23:18:18.619 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T23:18:19.119 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T23:18:19.128 INFO:teuthology.task.internal:Starting timer... 2024-07-26T23:18:19.128 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T23:18:19.192 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T23:18:19.213 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-07-26T23:18:19.213 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2024-07-26T23:18:19.305 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2024-07-26T23:18:19.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-07-26T23:18:19.385 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-07-26T23:18:19.388 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T23:18:19.388 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2024-07-26T23:18:19.417 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2024-07-26T23:18:19.417 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-07-26T23:18:19.459 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-07-26T23:18:19.459 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi139.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-07-26T23:18:19.460 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T23:18:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T23:18:19.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T23:18:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T23:18:19.525 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T23:18:19.525 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2024-07-26T23:18:19.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-07-26T23:18:19.603 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T23:18:19.679 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-26T23:18:19.683 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-26T23:18:27.167 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T23:18:28.109 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T23:18:28.111 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi139.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T23:24:08.092 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi139.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-07-26T23:24:08.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-07-26T23:24:08.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:24:08.189 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-07-26T23:24:08.266 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-07-26T23:24:08.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-07-26T23:24:08.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T23:24:08.343 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-07-26T23:24:08.426 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-07-26T23:24:08.426 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T23:24:08.448 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T23:24:08.448 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T23:24:08.449 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T23:24:08.452 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T23:24:08.452 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T23:24:08.478 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T23:24:08.492 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T23:24:08.496 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T23:24:08.514 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T23:24:08.517 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2024-07-26T23:24:08.527 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2024-07-26T23:24:08.536 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-07-26T23:24:08.541 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T23:24:08.547 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-07-26T23:24:08.555 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T23:24:08.562 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T23:24:08.575 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T23:24:08.610 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-07-26T23:24:08.613 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T23:24:08.613 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-07-26T23:24:08.614 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.614 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.614 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.614 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.634 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-07-26T23:24:08.637 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T23:24:08.637 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-07-26T23:24:08.637 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.637 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.637 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.637 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T23:24:08.638 INFO:teuthology.run_tasks:Running task install... 2024-07-26T23:24:08.647 DEBUG:teuthology.task.install:project ceph 2024-07-26T23:24:08.647 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b'}} 2024-07-26T23:24:08.647 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b'} 2024-07-26T23:24:08.647 INFO:teuthology.task.install:Using flavor: default 2024-07-26T23:24:08.656 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-07-26T23:24:08.656 INFO:teuthology.task.install:extra packages: [] 2024-07-26T23:24:08.657 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b', 'tag': None, 'wait_for_package': False} 2024-07-26T23:24:08.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=67f28c2a04ad1fe90ea612a39329cc6af2a92d8b 2024-07-26T23:24:08.659 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b', 'tag': None, 'wait_for_package': False} 2024-07-26T23:24:08.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=67f28c2a04ad1fe90ea612a39329cc6af2a92d8b 2024-07-26T23:24:08.802 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/ 2024-07-26T23:24:08.803 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5372.g67f28c2a 2024-07-26T23:24:08.812 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/ 2024-07-26T23:24:08.812 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5372.g67f28c2a 2024-07-26T23:24:08.914 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T23:24:08.914 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:24:08.914 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T23:24:08.924 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-07-26-0628/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T23:24:08.925 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:24:08.925 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T23:24:08.940 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-07-26T23:24:08.941 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T23:24:08.951 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-07-26T23:24:08.951 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T23:24:09.007 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T23:24:09.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T23:24:09.082 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T23:24:09.091 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T23:24:09.146 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2024-07-26T23:24:09.147 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2024-07-26T23:24:09.155 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-07-26T23:24:09.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-07-26T23:24:09.386 INFO:teuthology.orchestra.run.smithi177.stdout:45 files removed 2024-07-26T23:24:09.392 INFO:teuthology.orchestra.run.smithi139.stdout:45 files removed 2024-07-26T23:24:09.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2024-07-26T23:24:09.419 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw 2024-07-26T23:24:10.015 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 266 kB/s | 84 kB 00:00 2024-07-26T23:24:10.030 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 255 kB/s | 84 kB 00:00 2024-07-26T23:24:10.357 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 82 kB/s | 24 kB 00:00 2024-07-26T23:24:10.379 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 81 kB/s | 24 kB 00:00 2024-07-26T23:24:10.682 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-07-26T23:24:10.697 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 7.1 kB/s | 1.9 kB 00:00 2024-07-26T23:24:11.216 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-07-26T23:24:11.463 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-07-26T23:24:13.718 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 14 MB/s | 20 MB 00:01 2024-07-26T23:24:15.564 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 6.7 MB/s | 20 MB 00:02 2024-07-26T23:24:17.918 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-07-26T23:24:19.418 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 18 kB 00:00 2024-07-26T23:24:19.906 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-07-26T23:24:20.389 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-07-26T23:24:21.582 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 18 kB 00:00 2024-07-26T23:24:22.519 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-07-26T23:24:25.629 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-07-26T23:24:27.825 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-26T23:24:27.983 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:24:27.993 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 11 M 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 3.5 M 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 3.2 M 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 5.4 M 2024-07-26T23:24:27.994 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 22 M 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 25 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 758 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 511 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 5.5 M 2024-07-26T23:24:27.995 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 45 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 138 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 188 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 370 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 355 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 116 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T23:24:27.996 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:Install 30 Packages 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:27.997 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 62 M 2024-07-26T23:24:27.998 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:24:28.896 INFO:teuthology.orchestra.run.smithi177.stdout:(1/32): ceph-base-19.0.0-5372.g67f28c2a.el9.x86 12 MB/s | 5.4 MB 00:00 2024-07-26T23:24:28.946 INFO:teuthology.orchestra.run.smithi177.stdout:(2/32): ceph-selinux-19.0.0-5372.g67f28c2a.el9. 505 kB/s | 25 kB 00:00 2024-07-26T23:24:29.022 INFO:teuthology.orchestra.run.smithi177.stdout:(3/32): libcephfs2-19.0.0-5372.g67f28c2a.el9.x8 9.9 MB/s | 758 kB 00:00 2024-07-26T23:24:29.089 INFO:teuthology.orchestra.run.smithi177.stdout:(4/32): libradosstriper1-19.0.0-5372.g67f28c2a. 7.5 MB/s | 511 kB 00:00 2024-07-26T23:24:29.356 INFO:teuthology.orchestra.run.smithi177.stdout:(5/32): librgw2-19.0.0-5372.g67f28c2a.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-07-26T23:24:29.406 INFO:teuthology.orchestra.run.smithi177.stdout:(6/32): python3-ceph-argparse-19.0.0-5372.g67f2 904 kB/s | 45 kB 00:00 2024-07-26T23:24:29.473 INFO:teuthology.orchestra.run.smithi177.stdout:(7/32): python3-ceph-common-19.0.0-5372.g67f28c 2.0 MB/s | 138 kB 00:00 2024-07-26T23:24:29.673 INFO:teuthology.orchestra.run.smithi177.stdout:(8/32): ceph-common-19.0.0-5372.g67f28c2a.el9.x 18 MB/s | 22 MB 00:01 2024-07-26T23:24:29.698 INFO:teuthology.orchestra.run.smithi177.stdout:(9/32): python3-cephfs-19.0.0-5372.g67f28c2a.el 835 kB/s | 188 kB 00:00 2024-07-26T23:24:29.723 INFO:teuthology.orchestra.run.smithi177.stdout:(10/32): python3-rados-19.0.0-5372.g67f28c2a.el 7.2 MB/s | 370 kB 00:00 2024-07-26T23:24:29.749 INFO:teuthology.orchestra.run.smithi177.stdout:(11/32): python3-rbd-19.0.0-5372.g67f28c2a.el9. 6.9 MB/s | 355 kB 00:00 2024-07-26T23:24:29.774 INFO:teuthology.orchestra.run.smithi177.stdout:(12/32): python3-rgw-19.0.0-5372.g67f28c2a.el9. 2.3 MB/s | 116 kB 00:00 2024-07-26T23:24:29.908 INFO:teuthology.orchestra.run.smithi177.stdout:(13/32): ceph-radosgw-19.0.0-5372.g67f28c2a.el9 7.4 MB/s | 11 MB 00:01 2024-07-26T23:24:29.958 INFO:teuthology.orchestra.run.smithi177.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 512 kB/s | 107 kB 00:00 2024-07-26T23:24:30.013 INFO:teuthology.orchestra.run.smithi177.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 685 kB/s | 160 kB 00:00 2024-07-26T23:24:30.059 INFO:teuthology.orchestra.run.smithi177.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2024-07-26T23:24:30.188 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:24:30.192 INFO:teuthology.orchestra.run.smithi177.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 317 kB/s | 42 kB 00:00 2024-07-26T23:24:30.196 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:24:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:24:30.196 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 11 M 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 3.5 M 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 3.2 M 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 5.4 M 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 22 M 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 25 k 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T23:24:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 758 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 511 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 5.5 M 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T23:24:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 45 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 138 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 188 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 370 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 355 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 116 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T23:24:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:30.200 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:24:30.200 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:24:30.200 INFO:teuthology.orchestra.run.smithi139.stdout:Install 30 Packages 2024-07-26T23:24:30.200 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 2 Packages 2024-07-26T23:24:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:30.200 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 62 M 2024-07-26T23:24:30.200 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:24:30.218 INFO:teuthology.orchestra.run.smithi177.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-07-26T23:24:30.243 INFO:teuthology.orchestra.run.smithi177.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-07-26T23:24:30.319 INFO:teuthology.orchestra.run.smithi177.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 358 kB/s | 36 kB 00:00 2024-07-26T23:24:30.369 INFO:teuthology.orchestra.run.smithi177.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 378 kB/s | 47 kB 00:00 2024-07-26T23:24:30.394 INFO:teuthology.orchestra.run.smithi177.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-07-26T23:24:30.478 INFO:teuthology.orchestra.run.smithi177.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-07-26T23:24:30.504 INFO:teuthology.orchestra.run.smithi177.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-07-26T23:24:30.530 INFO:teuthology.orchestra.run.smithi177.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-07-26T23:24:30.554 INFO:teuthology.orchestra.run.smithi177.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-07-26T23:24:30.579 INFO:teuthology.orchestra.run.smithi177.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.6 MB/s | 192 kB 00:00 2024-07-26T23:24:30.629 INFO:teuthology.orchestra.run.smithi177.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-07-26T23:24:30.688 INFO:teuthology.orchestra.run.smithi177.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-07-26T23:24:30.755 INFO:teuthology.orchestra.run.smithi177.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-07-26T23:24:30.838 INFO:teuthology.orchestra.run.smithi177.stdout:(31/32): librados2-19.0.0-5372.g67f28c2a.el9.x8 17 MB/s | 3.5 MB 00:00 2024-07-26T23:24:30.989 INFO:teuthology.orchestra.run.smithi177.stdout:(32/32): librbd1-19.0.0-5372.g67f28c2a.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-07-26T23:24:30.994 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:24:30.994 INFO:teuthology.orchestra.run.smithi177.stdout:Total 21 MB/s | 62 MB 00:02 2024-07-26T23:24:31.093 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:24:31.128 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:24:31.128 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:24:31.467 INFO:teuthology.orchestra.run.smithi139.stdout:(1/32): ceph-radosgw-19.0.0-5372.g67f28c2a.el9. 13 MB/s | 11 MB 00:00 2024-07-26T23:24:31.559 INFO:teuthology.orchestra.run.smithi139.stdout:(2/32): ceph-base-19.0.0-5372.g67f28c2a.el9.x86 6.0 MB/s | 5.4 MB 00:00 2024-07-26T23:24:31.584 INFO:teuthology.orchestra.run.smithi139.stdout:(3/32): ceph-selinux-19.0.0-5372.g67f28c2a.el9. 216 kB/s | 25 kB 00:00 2024-07-26T23:24:31.651 INFO:teuthology.orchestra.run.smithi139.stdout:(4/32): libradosstriper1-19.0.0-5372.g67f28c2a. 7.5 MB/s | 511 kB 00:00 2024-07-26T23:24:31.686 INFO:teuthology.orchestra.run.smithi139.stdout:(5/32): libcephfs2-19.0.0-5372.g67f28c2a.el9.x8 5.9 MB/s | 758 kB 00:00 2024-07-26T23:24:31.702 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:24:31.702 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:24:31.752 INFO:teuthology.orchestra.run.smithi139.stdout:(6/32): python3-ceph-argparse-19.0.0-5372.g67f2 679 kB/s | 45 kB 00:00 2024-07-26T23:24:31.836 INFO:teuthology.orchestra.run.smithi139.stdout:(7/32): python3-ceph-common-19.0.0-5372.g67f28c 1.6 MB/s | 138 kB 00:00 2024-07-26T23:24:32.044 INFO:teuthology.orchestra.run.smithi139.stdout:(8/32): ceph-common-19.0.0-5372.g67f28c2a.el9.x 15 MB/s | 22 MB 00:01 2024-07-26T23:24:32.069 INFO:teuthology.orchestra.run.smithi139.stdout:(9/32): python3-cephfs-19.0.0-5372.g67f28c2a.el 805 kB/s | 188 kB 00:00 2024-07-26T23:24:32.095 INFO:teuthology.orchestra.run.smithi139.stdout:(10/32): python3-rados-19.0.0-5372.g67f28c2a.el 7.2 MB/s | 370 kB 00:00 2024-07-26T23:24:32.120 INFO:teuthology.orchestra.run.smithi139.stdout:(11/32): python3-rbd-19.0.0-5372.g67f28c2a.el9. 6.9 MB/s | 355 kB 00:00 2024-07-26T23:24:32.145 INFO:teuthology.orchestra.run.smithi139.stdout:(12/32): python3-rgw-19.0.0-5372.g67f28c2a.el9. 2.3 MB/s | 116 kB 00:00 2024-07-26T23:24:32.221 INFO:teuthology.orchestra.run.smithi139.stdout:(13/32): librgw2-19.0.0-5372.g67f28c2a.el9.x86_ 9.6 MB/s | 5.5 MB 00:00 2024-07-26T23:24:32.663 INFO:teuthology.orchestra.run.smithi139.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 197 kB/s | 107 kB 00:00 2024-07-26T23:24:32.705 INFO:teuthology.orchestra.run.smithi139.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 104 kB/s | 45 kB 00:00 2024-07-26T23:24:32.746 INFO:teuthology.orchestra.run.smithi139.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 267 kB/s | 160 kB 00:00 2024-07-26T23:24:32.997 INFO:teuthology.orchestra.run.smithi139.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 169 kB/s | 42 kB 00:00 2024-07-26T23:24:33.022 INFO:teuthology.orchestra.run.smithi139.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-07-26T23:24:33.047 INFO:teuthology.orchestra.run.smithi139.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 854 kB/s | 292 kB 00:00 2024-07-26T23:24:33.106 INFO:teuthology.orchestra.run.smithi139.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-07-26T23:24:33.107 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:24:33.123 INFO:teuthology.orchestra.run.smithi139.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 357 kB/s | 36 kB 00:00 2024-07-26T23:24:33.182 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-07-26T23:24:33.198 INFO:teuthology.orchestra.run.smithi139.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 315 kB/s | 47 kB 00:00 2024-07-26T23:24:33.210 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-07-26T23:24:33.236 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-07-26T23:24:33.257 INFO:teuthology.orchestra.run.smithi139.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2024-07-26T23:24:33.258 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-26T23:24:33.273 INFO:teuthology.orchestra.run.smithi139.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.4 MB/s | 49 kB 00:00 2024-07-26T23:24:33.283 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-26T23:24:33.290 INFO:teuthology.orchestra.run.smithi139.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 297 kB/s | 26 kB 00:00 2024-07-26T23:24:33.316 INFO:teuthology.orchestra.run.smithi139.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-07-26T23:24:33.349 INFO:teuthology.orchestra.run.smithi139.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.7 MB/s | 192 kB 00:00 2024-07-26T23:24:33.391 INFO:teuthology.orchestra.run.smithi139.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-07-26T23:24:33.485 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-26T23:24:33.492 INFO:teuthology.orchestra.run.smithi139.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-07-26T23:24:33.540 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-26T23:24:33.567 INFO:teuthology.orchestra.run.smithi139.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-07-26T23:24:33.767 INFO:teuthology.orchestra.run.smithi139.stdout:(31/32): librados2-19.0.0-5372.g67f28c2a.el9.x8 9.2 MB/s | 3.5 MB 00:00 2024-07-26T23:24:33.826 INFO:teuthology.orchestra.run.smithi139.stdout:(32/32): librbd1-19.0.0-5372.g67f28c2a.el9.x86_ 9.7 MB/s | 3.2 MB 00:00 2024-07-26T23:24:33.832 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:24:33.832 INFO:teuthology.orchestra.run.smithi139.stdout:Total 17 MB/s | 62 MB 00:03 2024-07-26T23:24:33.930 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:24:33.944 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-07-26T23:24:33.965 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:24:33.965 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:24:34.013 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 7/34 2024-07-26T23:24:34.123 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 7/34 2024-07-26T23:24:34.211 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 8/34 2024-07-26T23:24:34.239 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 9/34 2024-07-26T23:24:34.302 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 9/34 2024-07-26T23:24:34.339 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-07-26T23:24:34.410 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-5372.g67f28c2a.el9. 11/34 2024-07-26T23:24:34.478 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 12/34 2024-07-26T23:24:34.499 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_6 13/34 2024-07-26T23:24:34.552 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:24:34.552 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:24:34.564 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_6 13/34 2024-07-26T23:24:34.590 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-07-26T23:24:34.711 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-07-26T23:24:34.742 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-07-26T23:24:34.783 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-07-26T23:24:34.814 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-07-26T23:24:34.844 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-07-26T23:24:34.876 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-07-26T23:24:35.196 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-07-26T23:24:35.231 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 22/34 2024-07-26T23:24:35.327 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 22/34 2024-07-26T23:24:35.352 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 23/34 2024-07-26T23:24:35.396 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-07-26T23:24:35.887 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-07-26T23:24:35.917 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 26/34 2024-07-26T23:24:35.970 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:24:35.990 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 26/34 2024-07-26T23:24:36.070 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-07-26T23:24:36.076 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 27/34 2024-07-26T23:24:36.106 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-07-26T23:24:36.128 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-5372.g67f28c2a.el9.x8 28/34 2024-07-26T23:24:36.131 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-07-26T23:24:36.153 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-26T23:24:36.181 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-07-26T23:24:36.372 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-26T23:24:36.429 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-07-26T23:24:36.819 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-07-26T23:24:36.882 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 7/34 2024-07-26T23:24:36.970 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 7/34 2024-07-26T23:24:37.064 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 8/34 2024-07-26T23:24:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 9/34 2024-07-26T23:24:37.171 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 9/34 2024-07-26T23:24:37.217 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-07-26T23:24:37.264 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:19.0.0-5372.g67f28c2a.el9. 11/34 2024-07-26T23:24:37.332 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 12/34 2024-07-26T23:24:37.361 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_6 13/34 2024-07-26T23:24:37.425 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_6 13/34 2024-07-26T23:24:37.459 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-07-26T23:24:37.571 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-07-26T23:24:37.602 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-07-26T23:24:37.643 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-07-26T23:24:37.675 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-07-26T23:24:37.704 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-07-26T23:24:37.737 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-07-26T23:24:38.065 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-07-26T23:24:38.108 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 22/34 2024-07-26T23:24:38.206 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 22/34 2024-07-26T23:24:38.229 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 23/34 2024-07-26T23:24:38.265 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-07-26T23:24:38.419 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 29/34 2024-07-26T23:24:38.467 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 29/34 2024-07-26T23:24:38.755 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-07-26T23:24:38.777 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 26/34 2024-07-26T23:24:38.851 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 26/34 2024-07-26T23:24:38.925 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 27/34 2024-07-26T23:24:38.955 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:19.0.0-5372.g67f28c2a.el9.x8 28/34 2024-07-26T23:24:39.034 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 29/34 2024-07-26T23:24:39.071 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 30/34 2024-07-26T23:24:39.141 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 30/34 2024-07-26T23:24:39.142 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T23:24:39.142 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T23:24:39.142 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:39.173 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 31/34 2024-07-26T23:24:41.316 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 29/34 2024-07-26T23:24:41.360 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 29/34 2024-07-26T23:24:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 29/34 2024-07-26T23:24:41.971 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 30/34 2024-07-26T23:24:42.055 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 30/34 2024-07-26T23:24:42.055 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T23:24:42.055 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T23:24:42.055 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:42.090 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 31/34 2024-07-26T23:24:48.898 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 31/34 2024-07-26T23:24:48.898 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-07-26T23:24:48.898 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-07-26T23:24:48.898 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-07-26T23:24:48.898 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-07-26T23:24:48.898 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-07-26T23:24:48.899 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-07-26T23:24:48.899 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-07-26T23:24:48.899 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:49.918 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 32/34 2024-07-26T23:24:49.950 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 32/34 2024-07-26T23:24:49.950 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:24:49.950 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T23:24:49.950 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T23:24:49.950 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T23:24:49.950 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:49.952 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-26T23:24:50.037 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-26T23:24:50.037 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-26T23:24:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-26T23:24:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/34 2024-07-26T23:24:51.245 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/34 2024-07-26T23:24:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/34 2024-07-26T23:24:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/34 2024-07-26T23:24:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 5/34 2024-07-26T23:24:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_6 6/34 2024-07-26T23:24:51.246 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 7/34 2024-07-26T23:24:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5372.g67f28c2a.el9. 8/34 2024-07-26T23:24:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-5372.g67f28c2a.el9.x8 9/34 2024-07-26T23:24:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 10/34 2024-07-26T23:24:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 11/34 2024-07-26T23:24:51.247 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 12/34 2024-07-26T23:24:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 13/34 2024-07-26T23:24:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-07-26T23:24:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-07-26T23:24:51.248 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-07-26T23:24:51.249 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-07-26T23:24:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-07-26T23:24:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-07-26T23:24:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-07-26T23:24:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-07-26T23:24:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-07-26T23:24:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-07-26T23:24:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 31/34 2024-07-26T23:24:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-07-26T23:24:51.251 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 33/34 2024-07-26T23:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 31/34 2024-07-26T23:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2024-07-26T23:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2024-07-26T23:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2024-07-26T23:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2024-07-26T23:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2024-07-26T23:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /root 2024-07-26T23:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2024-07-26T23:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:52.112 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-07-26T23:24:52.113 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:52.113 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-07-26T23:24:52.113 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.113 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.113 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:24:52.114 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T23:24:52.114 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.114 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.114 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.114 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.114 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.115 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:52.116 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T23:24:52.117 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T23:24:52.117 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T23:24:52.117 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T23:24:52.117 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T23:24:52.117 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:52.117 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:24:52.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2024-07-26T23:24:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 32/34 2024-07-26T23:24:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 32/34 2024-07-26T23:24:52.898 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:24:52.898 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T23:24:52.898 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T23:24:52.898 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T23:24:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-26T23:24:52.991 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-07-26T23:24:52.991 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-26T23:24:53.021 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:28 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:24:53.148 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:24:53.149 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:24:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:24:53.149 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:24:53.149 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:24:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 50 M 2024-07-26T23:24:53.149 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:24:53.149 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 163 k 2024-07-26T23:24:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T23:24:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T23:24:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:53.150 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:24:53.150 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:24:53.150 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-07-26T23:24:53.150 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:24:53.151 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 51 M 2024-07-26T23:24:53.151 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 216 M 2024-07-26T23:24:53.152 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:24:53.416 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-07-26T23:24:53.450 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): libcephsqlite-19.0.0-5372.g67f28c2a.el9. 623 kB/s | 163 kB 00:00 2024-07-26T23:24:53.475 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-07-26T23:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 5/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_6 6/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 7/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5372.g67f28c2a.el9. 8/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:19.0.0-5372.g67f28c2a.el9.x8 9/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 10/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 11/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 12/34 2024-07-26T23:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 13/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-07-26T23:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 31/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-07-26T23:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 33/34 2024-07-26T23:24:54.767 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): ceph-test-19.0.0-5372.g67f28c2a.el9.x86_ 32 MB/s | 50 MB 00:01 2024-07-26T23:24:54.769 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:24:54.769 INFO:teuthology.orchestra.run.smithi177.stdout:Total 31 MB/s | 51 MB 00:01 2024-07-26T23:24:54.796 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:24:54.817 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:24:54.817 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:24:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T23:24:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T23:24:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T23:24:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T23:24:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:55.106 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:24:55.197 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:24:55.198 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:24:55.445 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-test 2024-07-26T23:24:55.727 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:24:55.826 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-07-26T23:24:55.882 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-07-26T23:24:55.911 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/4 2024-07-26T23:24:55.999 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:28 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:24:56.125 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:24:56.126 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:24:56.126 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:24:56.126 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:24:56.126 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:24:56.126 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 50 M 2024-07-26T23:24:56.126 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 163 k 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2024-07-26T23:24:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:24:56.128 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 51 M 2024-07-26T23:24:56.128 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 216 M 2024-07-26T23:24:56.128 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:24:56.316 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-07-26T23:24:56.358 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-07-26T23:24:56.574 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): libcephsqlite-19.0.0-5372.g67f28c2a.el9. 400 kB/s | 163 kB 00:00 2024-07-26T23:24:58.916 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): ceph-test-19.0.0-5372.g67f28c2a.el9.x86_ 18 MB/s | 50 MB 00:02 2024-07-26T23:24:58.918 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:24:58.919 INFO:teuthology.orchestra.run.smithi139.stdout:Total 18 MB/s | 51 MB 00:02 2024-07-26T23:24:58.945 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:24:58.966 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:24:58.966 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:24:59.345 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:24:59.345 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:24:59.884 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:24:59.979 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-07-26T23:25:00.015 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-07-26T23:25:00.035 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/4 2024-07-26T23:25:00.689 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/4 2024-07-26T23:25:00.767 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/4 2024-07-26T23:25:03.169 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/4 2024-07-26T23:25:03.169 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/4 2024-07-26T23:25:03.169 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/4 2024-07-26T23:25:03.169 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-07-26T23:25:03.436 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-07-26T23:25:03.436 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:03.436 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:25:03.436 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:03.436 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:03.436 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T23:25:03.436 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T23:25:03.437 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:03.437 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:25:03.701 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2024-07-26T23:25:04.266 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:39 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:25:04.404 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:25:04.407 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:25:04.407 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:04.407 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:25:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 6.6 k 2024-07-26T23:25:04.407 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:25:04.407 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 2.4 M 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.5 M 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 284 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 4.8 M 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 17 M 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T23:25:04.408 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T23:25:04.409 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:25:04.410 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:04.411 INFO:teuthology.orchestra.run.smithi177.stdout:Install 37 Packages 2024-07-26T23:25:04.411 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:04.411 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 30 M 2024-07-26T23:25:04.412 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 109 M 2024-07-26T23:25:04.412 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:25:04.829 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/4 2024-07-26T23:25:04.879 INFO:teuthology.orchestra.run.smithi177.stdout:(1/37): ceph-19.0.0-5372.g67f28c2a.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-07-26T23:25:04.925 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/4 2024-07-26T23:25:05.121 INFO:teuthology.orchestra.run.smithi177.stdout:(2/37): ceph-mds-19.0.0-5372.g67f28c2a.el9.x86_ 5.9 MB/s | 2.4 MB 00:00 2024-07-26T23:25:05.163 INFO:teuthology.orchestra.run.smithi177.stdout:(3/37): ceph-mgr-19.0.0-5372.g67f28c2a.el9.x86_ 3.4 MB/s | 1.5 MB 00:00 2024-07-26T23:25:05.247 INFO:teuthology.orchestra.run.smithi177.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5372.g67f2 3.3 MB/s | 284 kB 00:00 2024-07-26T23:25:05.322 INFO:teuthology.orchestra.run.smithi177.stdout:(5/37): ceph-mon-19.0.0-5372.g67f28c2a.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-07-26T23:25:05.497 INFO:teuthology.orchestra.run.smithi177.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-07-26T23:25:05.664 INFO:teuthology.orchestra.run.smithi177.stdout:(7/37): ceph-osd-19.0.0-5372.g67f28c2a.el9.x86_ 31 MB/s | 17 MB 00:00 2024-07-26T23:25:05.689 INFO:teuthology.orchestra.run.smithi177.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 556 kB/s | 106 kB 00:00 2024-07-26T23:25:05.756 INFO:teuthology.orchestra.run.smithi177.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-07-26T23:25:05.790 INFO:teuthology.orchestra.run.smithi177.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-07-26T23:25:05.815 INFO:teuthology.orchestra.run.smithi177.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-07-26T23:25:05.849 INFO:teuthology.orchestra.run.smithi177.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 734 kB/s | 135 kB 00:00 2024-07-26T23:25:06.115 INFO:teuthology.orchestra.run.smithi177.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-07-26T23:25:06.324 INFO:teuthology.orchestra.run.smithi177.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 338 kB/s | 172 kB 00:00 2024-07-26T23:25:06.391 INFO:teuthology.orchestra.run.smithi177.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 313 kB/s | 188 kB 00:00 2024-07-26T23:25:06.410 INFO:teuthology.orchestra.run.smithi177.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 264 kB/s | 77 kB 00:00 2024-07-26T23:25:06.433 INFO:teuthology.orchestra.run.smithi177.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 467 kB/s | 51 kB 00:00 2024-07-26T23:25:06.492 INFO:teuthology.orchestra.run.smithi177.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 413 kB/s | 42 kB 00:00 2024-07-26T23:25:06.534 INFO:teuthology.orchestra.run.smithi177.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-07-26T23:25:06.559 INFO:teuthology.orchestra.run.smithi177.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-07-26T23:25:06.601 INFO:teuthology.orchestra.run.smithi177.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2024-07-26T23:25:06.618 INFO:teuthology.orchestra.run.smithi177.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 736 kB/s | 11 kB 00:00 2024-07-26T23:25:06.643 INFO:teuthology.orchestra.run.smithi177.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-07-26T23:25:06.660 INFO:teuthology.orchestra.run.smithi177.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-07-26T23:25:06.677 INFO:teuthology.orchestra.run.smithi177.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 689 kB/s | 23 kB 00:00 2024-07-26T23:25:06.694 INFO:teuthology.orchestra.run.smithi177.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2024-07-26T23:25:06.711 INFO:teuthology.orchestra.run.smithi177.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 587 kB/s | 20 kB 00:00 2024-07-26T23:25:06.728 INFO:teuthology.orchestra.run.smithi177.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-07-26T23:25:06.753 INFO:teuthology.orchestra.run.smithi177.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-07-26T23:25:06.778 INFO:teuthology.orchestra.run.smithi177.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-07-26T23:25:06.795 INFO:teuthology.orchestra.run.smithi177.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-07-26T23:25:06.812 INFO:teuthology.orchestra.run.smithi177.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-07-26T23:25:06.838 INFO:teuthology.orchestra.run.smithi177.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-07-26T23:25:06.863 INFO:teuthology.orchestra.run.smithi177.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-07-26T23:25:06.880 INFO:teuthology.orchestra.run.smithi177.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-07-26T23:25:06.905 INFO:teuthology.orchestra.run.smithi177.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-07-26T23:25:06.930 INFO:teuthology.orchestra.run.smithi177.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-07-26T23:25:06.934 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:06.935 INFO:teuthology.orchestra.run.smithi177.stdout:Total 12 MB/s | 30 MB 00:02 2024-07-26T23:25:07.088 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:25:07.128 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:25:07.128 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:25:07.283 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/4 2024-07-26T23:25:07.284 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/4 2024-07-26T23:25:07.284 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/4 2024-07-26T23:25:07.284 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-07-26T23:25:07.574 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:25:07.577 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:25:07.594 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:07.595 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:07.853 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph 2024-07-26T23:25:08.432 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:41 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:08.567 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:08.570 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:08.570 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:25:08.570 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:08.570 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:25:08.570 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 6.6 k 2024-07-26T23:25:08.570 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 2.4 M 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.5 M 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 284 k 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 4.8 M 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 17 M 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T23:25:08.571 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T23:25:08.572 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T23:25:08.573 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:08.574 INFO:teuthology.orchestra.run.smithi139.stdout:Install 37 Packages 2024-07-26T23:25:08.575 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:08.576 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 30 M 2024-07-26T23:25:08.576 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 109 M 2024-07-26T23:25:08.576 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:25:08.877 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:25:08.941 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-07-26T23:25:08.979 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-07-26T23:25:09.010 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-07-26T23:25:09.080 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-07-26T23:25:09.143 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-07-26T23:25:09.176 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-07-26T23:25:09.267 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-07-26T23:25:09.345 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-07-26T23:25:09.386 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-07-26T23:25:09.421 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-07-26T23:25:09.451 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-07-26T23:25:09.486 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-07-26T23:25:09.519 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-07-26T23:25:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-07-26T23:25:09.586 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-07-26T23:25:09.620 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-07-26T23:25:09.669 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-07-26T23:25:09.709 INFO:teuthology.orchestra.run.smithi139.stdout:(1/37): ceph-19.0.0-5372.g67f28c2a.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-07-26T23:25:09.710 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-07-26T23:25:09.782 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-07-26T23:25:09.921 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-07-26T23:25:10.000 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-07-26T23:25:10.053 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-07-26T23:25:10.094 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-07-26T23:25:10.118 INFO:teuthology.orchestra.run.smithi139.stdout:(2/37): ceph-mds-19.0.0-5372.g67f28c2a.el9.x86_ 3.0 MB/s | 2.4 MB 00:00 2024-07-26T23:25:10.142 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-07-26T23:25:10.196 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-07-26T23:25:10.243 INFO:teuthology.orchestra.run.smithi139.stdout:(3/37): ceph-mgr-19.0.0-5372.g67f28c2a.el9.x86_ 1.7 MB/s | 1.5 MB 00:00 2024-07-26T23:25:10.335 INFO:teuthology.orchestra.run.smithi139.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5372.g67f2 3.0 MB/s | 284 kB 00:00 2024-07-26T23:25:10.382 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-07-26T23:25:10.430 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-07-26T23:25:10.508 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-07-26T23:25:10.535 INFO:teuthology.orchestra.run.smithi139.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-07-26T23:25:10.596 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-07-26T23:25:10.644 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-07-26T23:25:10.669 INFO:teuthology.orchestra.run.smithi139.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 9.2 MB/s | 1.2 MB 00:00 2024-07-26T23:25:10.735 INFO:teuthology.orchestra.run.smithi139.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-07-26T23:25:10.773 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-07-26T23:25:10.794 INFO:teuthology.orchestra.run.smithi139.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-07-26T23:25:10.861 INFO:teuthology.orchestra.run.smithi139.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-07-26T23:25:10.920 INFO:teuthology.orchestra.run.smithi139.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-07-26T23:25:10.935 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5372.g67f28c2a.el9. 32/37 2024-07-26T23:25:10.973 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 33/37 2024-07-26T23:25:10.998 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 33/37 2024-07-26T23:25:10.998 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:10.998 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T23:25:10.998 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T23:25:10.998 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T23:25:10.998 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:11.078 INFO:teuthology.orchestra.run.smithi139.stdout:(11/37): ceph-osd-19.0.0-5372.g67f28c2a.el9.x86 18 MB/s | 17 MB 00:00 2024-07-26T23:25:11.212 INFO:teuthology.orchestra.run.smithi139.stdout:(12/37): ceph-mon-19.0.0-5372.g67f28c2a.el9.x86 3.2 MB/s | 4.8 MB 00:01 2024-07-26T23:25:11.887 INFO:teuthology.orchestra.run.smithi139.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 51 kB/s | 35 kB 00:00 2024-07-26T23:25:11.912 INFO:teuthology.orchestra.run.smithi139.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 190 kB/s | 188 kB 00:00 2024-07-26T23:25:11.946 INFO:teuthology.orchestra.run.smithi139.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 198 kB/s | 172 kB 00:00 2024-07-26T23:25:12.450 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 34/37 2024-07-26T23:25:12.475 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 34/37 2024-07-26T23:25:12.476 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:12.476 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T23:25:12.476 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T23:25:12.476 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T23:25:12.477 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:12.604 INFO:teuthology.orchestra.run.smithi139.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 74 kB/s | 51 kB 00:00 2024-07-26T23:25:12.630 INFO:teuthology.orchestra.run.smithi139.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 104 kB/s | 77 kB 00:00 2024-07-26T23:25:12.655 INFO:teuthology.orchestra.run.smithi139.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 59 kB/s | 42 kB 00:00 2024-07-26T23:25:12.755 INFO:teuthology.orchestra.run.smithi139.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-07-26T23:25:12.789 INFO:teuthology.orchestra.run.smithi139.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-07-26T23:25:12.814 INFO:teuthology.orchestra.run.smithi139.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 107 kB/s | 22 kB 00:00 2024-07-26T23:25:12.839 INFO:teuthology.orchestra.run.smithi139.stdout:(22/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 236 kB/s | 43 kB 00:00 2024-07-26T23:25:12.889 INFO:teuthology.orchestra.run.smithi139.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-07-26T23:25:12.915 INFO:teuthology.orchestra.run.smithi139.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-07-26T23:25:12.916 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 35/37 2024-07-26T23:25:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 35/37 2024-07-26T23:25:12.940 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:12.940 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T23:25:12.941 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T23:25:12.941 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T23:25:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:12.941 INFO:teuthology.orchestra.run.smithi139.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 485 kB/s | 23 kB 00:00 2024-07-26T23:25:12.974 INFO:teuthology.orchestra.run.smithi139.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 585 kB/s | 19 kB 00:00 2024-07-26T23:25:12.999 INFO:teuthology.orchestra.run.smithi139.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-07-26T23:25:13.025 INFO:teuthology.orchestra.run.smithi139.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-07-26T23:25:13.049 INFO:teuthology.orchestra.run.smithi139.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-07-26T23:25:13.074 INFO:teuthology.orchestra.run.smithi139.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-07-26T23:25:13.100 INFO:teuthology.orchestra.run.smithi139.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-07-26T23:25:13.125 INFO:teuthology.orchestra.run.smithi139.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-07-26T23:25:13.151 INFO:teuthology.orchestra.run.smithi139.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-07-26T23:25:13.166 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 36/37 2024-07-26T23:25:13.176 INFO:teuthology.orchestra.run.smithi139.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-07-26T23:25:13.190 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 36/37 2024-07-26T23:25:13.190 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:13.190 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T23:25:13.190 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T23:25:13.190 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T23:25:13.190 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:13.193 INFO:teuthology.orchestra.run.smithi139.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-07-26T23:25:13.218 INFO:teuthology.orchestra.run.smithi139.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-07-26T23:25:13.243 INFO:teuthology.orchestra.run.smithi139.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-07-26T23:25:13.248 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:13.249 INFO:teuthology.orchestra.run.smithi139.stdout:Total 6.5 MB/s | 30 MB 00:04 2024-07-26T23:25:13.263 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 37/37 2024-07-26T23:25:13.397 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:25:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:25:13.436 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:25:13.880 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:25:13.881 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 37/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 5/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5372.g67f28c2a.el9. 6/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-07-26T23:25:14.971 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-07-26T23:25:14.972 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-07-26T23:25:14.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-07-26T23:25:15.189 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:25:15.254 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-07-26T23:25:15.291 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-07-26T23:25:15.323 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-07-26T23:25:15.396 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-07-26T23:25:15.455 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-07-26T23:25:15.488 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-07-26T23:25:15.580 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-07-26T23:25:15.650 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-07-26T23:25:15.691 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-07-26T23:25:15.726 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-07-26T23:25:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-07-26T23:25:15.790 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-07-26T23:25:15.823 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-07-26T23:25:15.864 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-07-26T23:25:15.899 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-07-26T23:25:15.933 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-07-26T23:25:15.982 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-07-26T23:25:16.013 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-07-26T23:25:16.055 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-07-26T23:25:16.055 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:16.055 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:25:16.055 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:16.055 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:16.055 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:16.056 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:16.056 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:16.056 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:16.056 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T23:25:16.056 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T23:25:16.056 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T23:25:16.056 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T23:25:16.057 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T23:25:16.058 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T23:25:16.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T23:25:16.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T23:25:16.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T23:25:16.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T23:25:16.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T23:25:16.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T23:25:16.059 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T23:25:16.060 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T23:25:16.060 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T23:25:16.060 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T23:25:16.060 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T23:25:16.060 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:16.060 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:25:16.087 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-07-26T23:25:16.227 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-07-26T23:25:16.313 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-07-26T23:25:16.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2024-07-26T23:25:16.366 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-07-26T23:25:16.405 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-07-26T23:25:16.454 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-07-26T23:25:16.509 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-07-26T23:25:16.696 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-07-26T23:25:16.750 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-07-26T23:25:16.829 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-07-26T23:25:16.905 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:51 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:25:16.918 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-07-26T23:25:16.965 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-07-26T23:25:16.991 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:25:17.032 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:25:17.033 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:25:17.033 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:25:17.083 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2024-07-26T23:25:17.093 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-07-26T23:25:17.265 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5372.g67f28c2a.el9. 32/37 2024-07-26T23:25:17.302 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 33/37 2024-07-26T23:25:17.328 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 33/37 2024-07-26T23:25:17.328 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:17.328 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T23:25:17.328 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T23:25:17.328 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T23:25:17.329 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:17.630 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:52 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:25:17.757 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:25:17.757 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 770 k 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 770 k 2024-07-26T23:25:17.758 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 777 k 2024-07-26T23:25:17.759 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:25:18.031 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-19.0.0-5372.g67f28c2a.el9.noarch.rpm 2.8 MB/s | 770 kB 00:00 2024-07-26T23:25:18.032 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:18.032 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.7 MB/s | 770 kB 00:00 2024-07-26T23:25:18.032 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:25:18.038 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:25:18.038 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:25:18.043 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:25:18.044 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:25:18.117 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:25:18.559 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:18.652 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:18.802 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 34/37 2024-07-26T23:25:18.829 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 34/37 2024-07-26T23:25:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T23:25:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T23:25:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T23:25:18.830 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:18.994 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:19.259 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:19.259 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:19.259 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:25:19.259 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:19.259 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:19.259 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:25:19.286 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 35/37 2024-07-26T23:25:19.310 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 35/37 2024-07-26T23:25:19.310 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:19.310 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T23:25:19.310 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T23:25:19.310 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T23:25:19.310 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:19.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2024-07-26T23:25:19.537 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 36/37 2024-07-26T23:25:19.560 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 36/37 2024-07-26T23:25:19.561 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:19.561 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T23:25:19.561 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T23:25:19.561 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T23:25:19.561 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:19.626 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 37/37 2024-07-26T23:25:20.032 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:55 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:25:20.165 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 145 k 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:20.166 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 145 k 2024-07-26T23:25:20.167 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 436 k 2024-07-26T23:25:20.167 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:25:20.370 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-19.0.0-5372.g67f28c 711 kB/s | 145 kB 00:00 2024-07-26T23:25:20.371 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:20.371 INFO:teuthology.orchestra.run.smithi177.stdout:Total 705 kB/s | 145 kB 00:00 2024-07-26T23:25:20.371 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:25:20.379 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:25:20.379 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:25:20.435 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:25:20.436 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:25:20.582 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:25:20.646 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a. 1/1 2024-07-26T23:25:20.672 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a. 1/1 2024-07-26T23:25:20.672 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:20.672 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T23:25:20.672 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:21.515 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 37/37 2024-07-26T23:25:21.516 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/37 2024-07-26T23:25:21.516 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/37 2024-07-26T23:25:21.516 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 3/37 2024-07-26T23:25:21.516 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 4/37 2024-07-26T23:25:21.516 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 5/37 2024-07-26T23:25:21.517 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5372.g67f28c2a.el9. 6/37 2024-07-26T23:25:21.517 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-07-26T23:25:21.517 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-07-26T23:25:21.517 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-07-26T23:25:21.517 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-07-26T23:25:21.517 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-07-26T23:25:21.517 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-07-26T23:25:21.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-07-26T23:25:21.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-07-26T23:25:21.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-07-26T23:25:21.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-07-26T23:25:21.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-07-26T23:25:21.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-07-26T23:25:21.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-07-26T23:25:21.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-07-26T23:25:21.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-07-26T23:25:21.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-07-26T23:25:21.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-07-26T23:25:21.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-07-26T23:25:21.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-07-26T23:25:21.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-07-26T23:25:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-07-26T23:25:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-07-26T23:25:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-07-26T23:25:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-07-26T23:25:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-07-26T23:25:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-07-26T23:25:21.521 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-07-26T23:25:21.521 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-07-26T23:25:21.521 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-07-26T23:25:21.521 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-07-26T23:25:21.543 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a. 1/1 2024-07-26T23:25:21.543 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:21.543 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:25:21.543 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:21.543 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:21.543 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:25:21.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2024-07-26T23:25:22.319 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:57 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:25:22.405 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:25:22.446 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:25:22.447 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:25:22.447 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:25:22.507 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-dashboard 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:22.597 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T23:25:22.598 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T23:25:22.599 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T23:25:22.600 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T23:25:22.600 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T23:25:22.600 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T23:25:22.600 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T23:25:22.600 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T23:25:22.600 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:22.600 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:22.920 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base 2024-07-26T23:25:23.052 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:58 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:25:23.180 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:25:23.182 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 14 M 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 34 k 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 16 k 2024-07-26T23:25:23.183 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:23.184 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:25:23.185 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:25:23.185 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-07-26T23:25:23.185 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:23.185 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 19 M 2024-07-26T23:25:23.185 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 154 M 2024-07-26T23:25:23.185 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:25:23.476 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:56 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:23.562 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-base-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:25:23.603 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:23.604 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:25:23.604 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:23.650 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install cephadm 2024-07-26T23:25:23.965 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): ceph-grafana-dashboards-19.0.0-5372.g67 176 kB/s | 34 kB 00:00 2024-07-26T23:25:24.002 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): ceph-prometheus-alerts-19.0.0-5372.g67f 72 kB/s | 16 kB 00:00 2024-07-26T23:25:24.124 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-07-26T23:25:24.175 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-07-26T23:25:24.193 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:57 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:24.242 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-07-26T23:25:24.267 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-07-26T23:25:24.319 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 770 k 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:25:24.320 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:24.321 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:25:24.321 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:24.321 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 770 k 2024-07-26T23:25:24.321 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 777 k 2024-07-26T23:25:24.321 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:25:24.325 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-07-26T23:25:24.351 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-07-26T23:25:24.376 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-07-26T23:25:24.401 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-07-26T23:25:24.443 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.6 MB/s | 863 kB 00:00 2024-07-26T23:25:24.620 INFO:teuthology.orchestra.run.smithi139.stdout:cephadm-19.0.0-5372.g67f28c2a.el9.noarch.rpm 2.5 MB/s | 770 kB 00:00 2024-07-26T23:25:24.621 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:24.621 INFO:teuthology.orchestra.run.smithi139.stdout:Total 2.5 MB/s | 770 kB 00:00 2024-07-26T23:25:24.621 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:25:24.626 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:25:24.626 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:25:24.632 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:25:24.632 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:25:24.705 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:25:24.752 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): ceph-mgr-dashboard-19.0.0-5372.g67f28c 14 MB/s | 14 MB 00:00 2024-07-26T23:25:24.755 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:24.755 INFO:teuthology.orchestra.run.smithi177.stdout:Total 12 MB/s | 19 MB 00:01 2024-07-26T23:25:24.804 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:25:24.822 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:25:24.822 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:25:25.076 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:25:25.077 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:25:25.131 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:25.249 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:25.605 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:25.665 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:25:25.761 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-07-26T23:25:25.795 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-07-26T23:25:25.842 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-07-26T23:25:25.873 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-07-26T23:25:25.875 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:25:25.875 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:25.875 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:25.875 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:25.876 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:25.876 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:25.971 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-07-26T23:25:26.064 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-immutable-object-cache 2024-07-26T23:25:26.110 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-07-26T23:25:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-07-26T23:25:26.276 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-07-26T23:25:26.309 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-07-26T23:25:26.341 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5372.g67f28c2a.el9 10/12 2024-07-26T23:25:26.620 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:59 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:26.749 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:26.750 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:26.750 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:25:26.750 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:26.750 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:25:26.750 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 145 k 2024-07-26T23:25:26.750 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:26.750 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:25:26.751 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:26.751 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:25:26.751 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:26.752 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 145 k 2024-07-26T23:25:26.752 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 436 k 2024-07-26T23:25:26.752 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:25:26.968 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-immutable-object-cache-19.0.0-5372.g67f28c 668 kB/s | 145 kB 00:00 2024-07-26T23:25:26.969 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:26.969 INFO:teuthology.orchestra.run.smithi139.stdout:Total 663 kB/s | 145 kB 00:00 2024-07-26T23:25:26.969 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:25:26.977 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:25:26.977 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:25:27.033 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:25:27.033 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:25:27.178 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:25:27.252 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a. 1/1 2024-07-26T23:25:27.277 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a. 1/1 2024-07-26T23:25:27.277 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:25:27.277 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T23:25:27.277 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:28.222 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a. 1/1 2024-07-26T23:25:28.223 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:28.223 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:28.223 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:25:28.223 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:28.223 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:28.421 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr 2024-07-26T23:25:28.983 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:01 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:29.076 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-mgr-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:25:29.118 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:29.119 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:25:29.120 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:29.182 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-dashboard 2024-07-26T23:25:29.478 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5372.g67f28c2a.el 11/12 2024-07-26T23:25:29.537 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noa 12/12 2024-07-26T23:25:29.725 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:02 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:29.854 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:29.855 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:29.855 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:25:29.855 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:29.855 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:25:29.855 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 14 M 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 34 k 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 16 k 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T23:25:29.856 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T23:25:29.857 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T23:25:29.857 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:29.857 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:25:29.857 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:29.857 INFO:teuthology.orchestra.run.smithi139.stdout:Install 12 Packages 2024-07-26T23:25:29.857 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:29.858 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 19 M 2024-07-26T23:25:29.858 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 154 M 2024-07-26T23:25:29.858 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:25:30.347 INFO:teuthology.orchestra.run.smithi139.stdout:(1/12): ceph-prometheus-alerts-19.0.0-5372.g67f 83 kB/s | 16 kB 00:00 2024-07-26T23:25:30.364 INFO:teuthology.orchestra.run.smithi139.stdout:(2/12): ceph-grafana-dashboards-19.0.0-5372.g67 159 kB/s | 34 kB 00:00 2024-07-26T23:25:30.555 INFO:teuthology.orchestra.run.smithi139.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-07-26T23:25:30.731 INFO:teuthology.orchestra.run.smithi139.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-07-26T23:25:30.798 INFO:teuthology.orchestra.run.smithi139.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-07-26T23:25:30.823 INFO:teuthology.orchestra.run.smithi139.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 788 kB/s | 19 kB 00:00 2024-07-26T23:25:30.856 INFO:teuthology.orchestra.run.smithi139.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-07-26T23:25:30.915 INFO:teuthology.orchestra.run.smithi139.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-07-26T23:25:30.940 INFO:teuthology.orchestra.run.smithi139.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-07-26T23:25:30.974 INFO:teuthology.orchestra.run.smithi139.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-07-26T23:25:31.024 INFO:teuthology.orchestra.run.smithi139.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-07-26T23:25:31.158 INFO:teuthology.orchestra.run.smithi139.stdout:(12/12): ceph-mgr-dashboard-19.0.0-5372.g67f28c 14 MB/s | 14 MB 00:01 2024-07-26T23:25:31.162 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:31.162 INFO:teuthology.orchestra.run.smithi139.stdout:Total 15 MB/s | 19 MB 00:01 2024-07-26T23:25:31.211 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:25:31.229 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:25:31.229 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:25:31.272 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noa 12/12 2024-07-26T23:25:31.272 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5372.g67f28c2a.el 1/12 2024-07-26T23:25:31.272 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noa 2/12 2024-07-26T23:25:31.272 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5372.g67f28c2a.el9 3/12 2024-07-26T23:25:31.272 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-07-26T23:25:31.272 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-07-26T23:25:31.272 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-07-26T23:25:31.273 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-07-26T23:25:31.273 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-07-26T23:25:31.273 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-07-26T23:25:31.273 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-07-26T23:25:31.273 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-07-26T23:25:31.467 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:25:31.467 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:25:31.573 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-07-26T23:25:31.573 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:31.573 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T23:25:31.574 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T23:25:31.575 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:31.575 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:25:31.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-diskprediction-local 2024-07-26T23:25:32.062 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:25:32.142 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-07-26T23:25:32.185 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-07-26T23:25:32.232 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-07-26T23:25:32.262 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-07-26T23:25:32.361 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:07 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:25:32.366 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-07-26T23:25:32.489 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:25:32.490 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-07-26T23:25:32.490 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:25:32.490 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-07-26T23:25:32.490 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:25:32.490 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 7.4 M 2024-07-26T23:25:32.490 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T23:25:32.491 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:32.492 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:25:32.492 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-07-26T23:25:32.492 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-07-26T23:25:32.492 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:25:32.492 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 38 M 2024-07-26T23:25:32.492 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 209 M 2024-07-26T23:25:32.492 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:25:32.537 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-07-26T23:25:32.628 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-07-26T23:25:32.689 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-07-26T23:25:32.717 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-07-26T23:25:32.749 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5372.g67f28c2a.el9 10/12 2024-07-26T23:25:33.185 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-53 16 MB/s | 7.4 MB 00:00 2024-07-26T23:25:33.227 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 390 kB/s | 195 kB 00:00 2024-07-26T23:25:33.444 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-07-26T23:25:33.469 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-07-26T23:25:33.536 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 198 kB/s | 18 kB 00:00 2024-07-26T23:25:33.620 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 234 kB/s | 35 kB 00:00 2024-07-26T23:25:33.728 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-07-26T23:25:33.862 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2024-07-26T23:25:33.970 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-07-26T23:25:35.371 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.6 MB/s | 4.7 MB 00:01 2024-07-26T23:25:35.870 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5372.g67f28c2a.el 11/12 2024-07-26T23:25:35.912 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.3 MB/s | 5.1 MB 00:02 2024-07-26T23:25:35.928 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noa 12/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noa 12/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5372.g67f28c2a.el 1/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noa 2/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5372.g67f28c2a.el9 3/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-07-26T23:25:37.831 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-07-26T23:25:37.832 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-07-26T23:25:37.832 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-07-26T23:25:37.832 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T23:25:38.156 INFO:teuthology.orchestra.run.smithi139.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T23:25:38.157 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T23:25:38.157 INFO:teuthology.orchestra.run.smithi139.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T23:25:38.157 INFO:teuthology.orchestra.run.smithi139.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T23:25:38.157 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T23:25:38.157 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T23:25:38.157 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:38.157 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:38.407 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-diskprediction-local 2024-07-26T23:25:38.960 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:11 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:39.095 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:39.097 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-07-26T23:25:39.097 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:25:39.097 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-07-26T23:25:39.097 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:25:39.097 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 7.4 M 2024-07-26T23:25:39.097 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T23:25:39.098 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:39.099 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:25:39.099 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-07-26T23:25:39.099 INFO:teuthology.orchestra.run.smithi139.stdout:Install 12 Packages 2024-07-26T23:25:39.099 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:39.099 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 38 M 2024-07-26T23:25:39.099 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 209 M 2024-07-26T23:25:39.100 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:25:39.811 INFO:teuthology.orchestra.run.smithi139.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-53 16 MB/s | 7.4 MB 00:00 2024-07-26T23:25:39.852 INFO:teuthology.orchestra.run.smithi139.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 386 kB/s | 195 kB 00:00 2024-07-26T23:25:39.903 INFO:teuthology.orchestra.run.smithi139.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-07-26T23:25:39.945 INFO:teuthology.orchestra.run.smithi139.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-07-26T23:25:39.995 INFO:teuthology.orchestra.run.smithi139.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-07-26T23:25:40.120 INFO:teuthology.orchestra.run.smithi139.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-07-26T23:25:40.146 INFO:teuthology.orchestra.run.smithi139.stdout:(7/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-07-26T23:25:40.171 INFO:teuthology.orchestra.run.smithi139.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.7 MB/s | 244 kB 00:00 2024-07-26T23:25:40.213 INFO:teuthology.orchestra.run.smithi139.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-07-26T23:25:40.288 INFO:teuthology.orchestra.run.smithi139.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 16 MB/s | 4.7 MB 00:00 2024-07-26T23:25:40.605 INFO:teuthology.orchestra.run.smithi139.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-07-26T23:25:40.689 INFO:teuthology.orchestra.run.smithi139.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.4 MB/s | 5.1 MB 00:00 2024-07-26T23:25:40.689 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:40.689 INFO:teuthology.orchestra.run.smithi139.stdout:Total 24 MB/s | 38 MB 00:01 2024-07-26T23:25:41.038 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:25:41.064 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:25:41.065 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:25:41.500 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:25:41.500 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:25:42.294 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:25:42.365 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-07-26T23:25:42.412 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-07-26T23:25:42.510 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-07-26T23:25:42.546 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-07-26T23:25:42.724 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-07-26T23:25:42.761 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-07-26T23:25:42.838 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-07-26T23:25:43.485 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-07-26T23:25:43.615 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-07-26T23:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-07-26T23:25:45.879 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-07-26T23:25:45.910 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28 12/12 2024-07-26T23:25:48.491 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28 12/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28 1/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-07-26T23:25:48.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-07-26T23:25:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-07-26T23:25:48.917 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:48.917 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T23:25:48.918 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T23:25:48.919 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T23:25:48.919 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:48.919 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:49.281 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-rook 2024-07-26T23:25:49.846 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:22 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:49.977 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 49 k 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T23:25:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T23:25:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T23:25:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T23:25:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T23:25:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T23:25:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T23:25:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T23:25:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T23:25:49.982 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T23:25:49.982 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T23:25:49.982 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:49.982 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:25:49.983 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:49.983 INFO:teuthology.orchestra.run.smithi139.stdout:Install 13 Packages 2024-07-26T23:25:49.983 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:49.984 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 2.2 M 2024-07-26T23:25:49.984 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 26 M 2024-07-26T23:25:49.984 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:25:50.252 INFO:teuthology.orchestra.run.smithi139.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 26 kB 00:00 2024-07-26T23:25:50.277 INFO:teuthology.orchestra.run.smithi139.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 19 kB 00:00 2024-07-26T23:25:50.302 INFO:teuthology.orchestra.run.smithi139.stdout:(3/13): ceph-mgr-rook-19.0.0-5372.g67f28c2a.el9 279 kB/s | 49 kB 00:00 2024-07-26T23:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-07-26T23:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-07-26T23:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-07-26T23:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-07-26T23:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-07-26T23:25:50.620 INFO:teuthology.orchestra.run.smithi139.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 80 kB/s | 14 kB 00:00 2024-07-26T23:25:50.653 INFO:teuthology.orchestra.run.smithi139.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2024-07-26T23:25:50.679 INFO:teuthology.orchestra.run.smithi139.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-07-26T23:25:50.704 INFO:teuthology.orchestra.run.smithi139.stdout:(12/13): python3-google-auth-2.32.0-1.el9.noarc 1.3 MB/s | 216 kB 00:00 2024-07-26T23:25:50.738 INFO:teuthology.orchestra.run.smithi139.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-07-26T23:25:50.740 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:50.740 INFO:teuthology.orchestra.run.smithi139.stdout:Total 2.9 MB/s | 2.2 MB 00:00 2024-07-26T23:25:50.803 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:25:50.818 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:25:50.818 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:25:50.907 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:25:50.908 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:25:51.415 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:25:51.600 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-07-26T23:25:51.655 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-07-26T23:25:51.711 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-07-26T23:25:51.748 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-07-26T23:25:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-07-26T23:25:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-07-26T23:25:52.066 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-07-26T23:25:52.115 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-07-26T23:25:52.793 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 884 kB/s | 16 MB 00:18 2024-07-26T23:25:52.795 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:52.795 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.9 MB/s | 38 MB 00:20 2024-07-26T23:25:52.819 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-07-26T23:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-07-26T23:25:52.922 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-07-26T23:25:52.969 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-07-26T23:25:52.991 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 13/13 2024-07-26T23:25:53.149 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:25:53.175 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:25:53.175 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:25:53.610 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:25:53.610 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:25:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 13/13 2024-07-26T23:25:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 1/13 2024-07-26T23:25:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-07-26T23:25:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-07-26T23:25:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-07-26T23:25:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-07-26T23:25:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-07-26T23:25:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-07-26T23:25:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-07-26T23:25:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-07-26T23:25:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-07-26T23:25:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-07-26T23:25:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-07-26T23:25:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-07-26T23:25:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:53.964 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:53.964 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:53.965 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:54.237 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-cephadm 2024-07-26T23:25:54.360 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:25:54.434 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-07-26T23:25:54.475 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-07-26T23:25:54.574 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-07-26T23:25:54.619 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-07-26T23:25:54.781 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:27 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:25:54.796 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-07-26T23:25:54.825 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-07-26T23:25:54.909 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-07-26T23:25:54.910 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:25:54.911 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:54.911 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:25:54.911 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:54.911 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 150 k 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2024-07-26T23:25:54.912 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:54.913 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 7.0 M 2024-07-26T23:25:54.913 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 33 M 2024-07-26T23:25:54.913 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:25:55.566 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-07-26T23:25:55.689 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-07-26T23:25:55.694 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-07-26T23:25:55.714 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): ceph-mgr-cephadm-19.0.0-5372.g67f28c2a.e 591 kB/s | 150 kB 00:00 2024-07-26T23:25:55.815 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.3 MB/s | 548 kB 00:00 2024-07-26T23:25:55.840 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 460 kB/s | 58 kB 00:00 2024-07-26T23:25:55.865 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-07-26T23:25:56.930 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-07-26T23:25:57.190 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-07-26T23:25:57.192 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:25:57.193 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.1 MB/s | 7.0 MB 00:02 2024-07-26T23:25:57.264 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:25:57.275 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:25:57.275 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:25:57.372 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:25:57.374 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:25:57.675 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:25:57.847 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-07-26T23:25:57.900 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-07-26T23:25:57.966 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-07-26T23:25:58.000 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28 12/12 2024-07-26T23:25:58.417 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-07-26T23:25:58.510 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-07-26T23:25:58.584 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-07-26T23:25:58.611 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 6/6 2024-07-26T23:25:59.352 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 6/6 2024-07-26T23:25:59.352 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/6 2024-07-26T23:25:59.352 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-07-26T23:25:59.352 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-07-26T23:25:59.352 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-07-26T23:25:59.352 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-07-26T23:25:59.637 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-07-26T23:25:59.637 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:59.637 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:25:59.637 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:25:59.637 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T23:25:59.637 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T23:25:59.638 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T23:25:59.638 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T23:25:59.638 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T23:25:59.638 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:25:59.638 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:25:59.893 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-fuse 2024-07-26T23:26:00.435 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:33 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:00.462 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28 12/12 2024-07-26T23:26:00.462 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28 1/12 2024-07-26T23:26:00.462 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-07-26T23:26:00.463 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-07-26T23:26:00.563 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:00.564 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:00.564 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:26:00.564 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:00.564 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:00.564 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 904 k 2024-07-26T23:26:00.564 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:26:00.564 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T23:26:00.565 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:00.565 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:00.565 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:00.565 INFO:teuthology.orchestra.run.smithi139.stdout:Install 2 Packages 2024-07-26T23:26:00.565 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:00.565 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 984 k 2024-07-26T23:26:00.565 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 2.9 M 2024-07-26T23:26:00.566 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:00.881 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-07-26T23:26:00.881 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:00.881 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:00.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:26:00.881 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T23:26:00.881 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:00.882 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:00.961 INFO:teuthology.orchestra.run.smithi139.stdout:(1/2): ceph-fuse-19.0.0-5372.g67f28c2a.el9.x86_ 3.2 MB/s | 904 kB 00:00 2024-07-26T23:26:01.111 INFO:teuthology.orchestra.run.smithi139.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 186 kB/s | 80 kB 00:00 2024-07-26T23:26:01.112 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:01.112 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.8 MB/s | 984 kB 00:00 2024-07-26T23:26:01.124 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:01.147 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:01.147 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:01.225 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:01.225 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:01.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-rook 2024-07-26T23:26:01.429 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:01.573 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-07-26T23:26:01.645 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/2 2024-07-26T23:26:01.853 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:36 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:01.982 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:01.983 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 49 k 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T23:26:01.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2024-07-26T23:26:01.985 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:01.986 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.2 M 2024-07-26T23:26:01.986 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2024-07-26T23:26:01.986 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:02.296 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/2 2024-07-26T23:26:02.297 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/2 2024-07-26T23:26:02.533 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): ceph-mgr-rook-19.0.0-5372.g67f28c2a.el9 267 kB/s | 49 kB 00:00 2024-07-26T23:26:02.551 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-07-26T23:26:02.552 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:02.552 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:02.552 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-07-26T23:26:02.552 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:02.552 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:02.608 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 99 kB/s | 26 kB 00:00 2024-07-26T23:26:02.625 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 69 kB/s | 19 kB 00:00 2024-07-26T23:26:02.774 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-volume 2024-07-26T23:26:02.950 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-07-26T23:26:03.050 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 656 kB/s | 279 kB 00:00 2024-07-26T23:26:03.067 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 458 kB/s | 54 kB 00:00 2024-07-26T23:26:03.092 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 397 kB/s | 222 kB 00:00 2024-07-26T23:26:03.193 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 226 kB/s | 32 kB 00:00 2024-07-26T23:26:03.218 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 94 kB/s | 14 kB 00:00 2024-07-26T23:26:03.243 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-26T23:26:03.260 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): python3-google-auth-2.32.0-1.el9.noarc 1.8 MB/s | 216 kB 00:00 2024-07-26T23:26:03.286 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-07-26T23:26:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.2 MB/s | 1.0 MB 00:00 2024-07-26T23:26:03.322 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:03.322 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-07-26T23:26:03.338 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:36 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:03.385 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:03.399 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:03.400 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:03.473 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:03.473 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:03.473 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-07-26T23:26:03.473 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 274 k 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 274 k 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 1.3 M 2024-07-26T23:26:03.474 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:03.485 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:03.486 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:03.710 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-volume-19.0.0-5372.g67f28c2a.el9.noarch.rp 1.1 MB/s | 274 kB 00:00 2024-07-26T23:26:03.710 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:03.711 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.1 MB/s | 274 kB 00:00 2024-07-26T23:26:03.711 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:03.719 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:03.720 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:03.750 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:03.751 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:03.967 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:04.004 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:04.070 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:26:04.086 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:26:04.087 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:26:04.087 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T23:26:04.087 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:04.273 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-07-26T23:26:04.337 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-07-26T23:26:04.400 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-07-26T23:26:04.438 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-07-26T23:26:04.496 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-07-26T23:26:04.593 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-07-26T23:26:04.707 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-07-26T23:26:04.764 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-07-26T23:26:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:26:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:05.045 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:26:05.045 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:05.045 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:05.338 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados-devel 2024-07-26T23:26:05.476 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-07-26T23:26:05.528 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-07-26T23:26:05.561 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-07-26T23:26:05.610 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-07-26T23:26:05.631 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 13/13 2024-07-26T23:26:05.903 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:38 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:06.035 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 127 k 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:06.036 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:26:06.037 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:06.037 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 127 k 2024-07-26T23:26:06.037 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 456 k 2024-07-26T23:26:06.037 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:06.274 INFO:teuthology.orchestra.run.smithi139.stdout:librados-devel-19.0.0-5372.g67f28c2a.el9.x86_64 534 kB/s | 127 kB 00:00 2024-07-26T23:26:06.275 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:06.275 INFO:teuthology.orchestra.run.smithi139.stdout:Total 530 kB/s | 127 kB 00:00 2024-07-26T23:26:06.275 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:06.281 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:06.281 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 13/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 1/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-07-26T23:26:06.292 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-07-26T23:26:06.293 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-07-26T23:26:06.293 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-07-26T23:26:06.293 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-07-26T23:26:06.293 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-07-26T23:26:06.345 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:06.345 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:06.495 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:06.599 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:06.611 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-07-26T23:26:06.611 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:06.611 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:06.611 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:26:06.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T23:26:06.611 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:06.612 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:06.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-cephadm 2024-07-26T23:26:07.042 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:07.319 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:07.319 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:07.319 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:07.319 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:07.319 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:07.320 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:07.414 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:42 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:07.543 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:07.543 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:07.543 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:26:07.543 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 150 k 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:07.544 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-07-26T23:26:07.545 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:07.545 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 7.0 M 2024-07-26T23:26:07.545 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 33 M 2024-07-26T23:26:07.545 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:07.588 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs2 2024-07-26T23:26:07.974 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-07-26T23:26:08.021 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): ceph-mgr-cephadm-19.0.0-5372.g67f28c2a.e 630 kB/s | 150 kB 00:00 2024-07-26T23:26:08.080 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-07-26T23:26:08.105 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.2 MB/s | 51 kB 00:00 2024-07-26T23:26:08.130 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 541 kB/s | 58 kB 00:00 2024-07-26T23:26:08.151 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:41 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:08.241 INFO:teuthology.orchestra.run.smithi139.stdout:Package libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:08.289 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:08.290 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:26:08.290 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:08.363 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs-devel 2024-07-26T23:26:08.664 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.8 MB/s | 6.0 MB 00:00 2024-07-26T23:26:08.666 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:08.666 INFO:teuthology.orchestra.run.smithi177.stdout:Total 6.2 MB/s | 7.0 MB 00:01 2024-07-26T23:26:08.737 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:08.748 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:08.749 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:08.848 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:08.848 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:08.937 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:41 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:09.071 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:09.072 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:09.072 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:26:09.073 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:09.073 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:09.073 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 32 k 2024-07-26T23:26:09.073 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:09.073 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:09.073 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:09.073 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:26:09.074 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:09.074 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 32 k 2024-07-26T23:26:09.074 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 142 k 2024-07-26T23:26:09.074 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:09.149 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:09.252 INFO:teuthology.orchestra.run.smithi139.stdout:libcephfs-devel-19.0.0-5372.g67f28c2a.el9.x86_6 178 kB/s | 32 kB 00:00 2024-07-26T23:26:09.253 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:09.253 INFO:teuthology.orchestra.run.smithi139.stdout:Total 176 kB/s | 32 kB 00:00 2024-07-26T23:26:09.253 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:09.256 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:09.256 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:09.273 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:09.274 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:09.319 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-07-26T23:26:09.374 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:09.382 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-07-26T23:26:09.508 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:09.888 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:09.901 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-07-26T23:26:10.008 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-07-26T23:26:10.078 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-07-26T23:26:10.102 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 6/6 2024-07-26T23:26:10.117 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:10.117 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:10.117 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:10.117 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:10.117 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:10.118 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:10.316 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados2 2024-07-26T23:26:10.863 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:43 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 6/6 2024-07-26T23:26:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 1/6 2024-07-26T23:26:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-07-26T23:26:10.878 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-07-26T23:26:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-07-26T23:26:10.879 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-07-26T23:26:10.948 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:10.992 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:10.992 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:26:10.993 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:11.067 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librbd1 2024-07-26T23:26:11.141 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-07-26T23:26:11.141 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:11.141 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:11.141 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:26:11.142 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T23:26:11.142 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T23:26:11.142 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T23:26:11.142 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T23:26:11.142 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T23:26:11.142 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:11.142 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:11.389 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2024-07-26T23:26:11.617 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:44 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:11.702 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:11.746 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:11.747 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:26:11.747 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:11.820 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rados 2024-07-26T23:26:11.932 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:46 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:12.063 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:12.063 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:12.063 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:26:12.063 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:12.063 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 904 k 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 984 k 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.9 M 2024-07-26T23:26:12.064 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:12.261 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 492 kB/s | 80 kB 00:00 2024-07-26T23:26:12.367 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:45 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:12.386 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-19.0.0-5372.g67f28c2a.el9.x86_ 3.1 MB/s | 904 kB 00:00 2024-07-26T23:26:12.387 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:12.387 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.0 MB/s | 984 kB 00:00 2024-07-26T23:26:12.398 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:12.420 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:12.420 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:12.452 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:12.495 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:12.496 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:26:12.496 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:12.499 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:12.499 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:12.570 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rgw 2024-07-26T23:26:12.694 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:12.839 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-07-26T23:26:12.911 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/2 2024-07-26T23:26:13.114 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:46 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:13.200 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:13.242 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:13.243 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:26:13.243 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:13.313 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-cephfs 2024-07-26T23:26:13.593 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/2 2024-07-26T23:26:13.593 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/2 2024-07-26T23:26:13.859 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:46 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:13.882 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-07-26T23:26:13.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:13.883 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:13.883 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-07-26T23:26:13.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:13.883 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:13.945 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:13.988 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:13.989 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:26:13.989 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:14.069 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rbd 2024-07-26T23:26:14.173 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-volume 2024-07-26T23:26:14.610 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:47 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:14.696 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:14.734 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:49 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:14.738 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:14.739 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:26:14.739 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:14.825 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-fuse 2024-07-26T23:26:14.866 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-5372.g67f28c2a.el9 ceph-noarch 274 k 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:14.867 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 274 k 2024-07-26T23:26:14.868 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.3 M 2024-07-26T23:26:14.868 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:15.119 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-volume-19.0.0-5372.g67f28c2a.el9.noarch.rp 1.1 MB/s | 274 kB 00:00 2024-07-26T23:26:15.120 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:15.120 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.1 MB/s | 274 kB 00:00 2024-07-26T23:26:15.120 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:15.130 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:15.130 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:15.154 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:15.154 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:15.365 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:15.371 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:48 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:26:15.445 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:26:15.446 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:26:15.446 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T23:26:15.446 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:15.500 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:15.501 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:15.501 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-07-26T23:26:15.501 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:15.501 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:15.501 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 85 k 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 85 k 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 231 k 2024-07-26T23:26:15.502 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:15.704 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-fuse-19.0.0-5372.g67f28c2a.el9.x86_64.rpm 424 kB/s | 85 kB 00:00 2024-07-26T23:26:15.705 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:15.705 INFO:teuthology.orchestra.run.smithi139.stdout:Total 420 kB/s | 85 kB 00:00 2024-07-26T23:26:15.705 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:15.712 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:15.712 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:15.779 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:15.780 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:15.924 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:16.011 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:16.389 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 1/1 2024-07-26T23:26:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:16.456 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-5372.g67f28c2a.el9.noarch 2024-07-26T23:26:16.456 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:16.456 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:16.641 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:16.641 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:16.641 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:16.641 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:16.641 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:16.641 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:16.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2024-07-26T23:26:16.882 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-mirror 2024-07-26T23:26:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:52 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:17.437 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:50 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:17.459 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:17.459 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 127 k 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:26:17.460 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:17.461 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 127 k 2024-07-26T23:26:17.461 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 456 k 2024-07-26T23:26:17.461 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:17.566 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:17.567 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:17.567 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:26:17.567 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:17.567 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:17.567 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 3.2 M 2024-07-26T23:26:17.567 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:17.568 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:17.568 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:17.568 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:26:17.568 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:17.568 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.2 M 2024-07-26T23:26:17.568 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 14 M 2024-07-26T23:26:17.568 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:17.658 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-19.0.0-5372.g67f28c2a.el9.x86_64 642 kB/s | 127 kB 00:00 2024-07-26T23:26:17.659 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:17.659 INFO:teuthology.orchestra.run.smithi177.stdout:Total 637 kB/s | 127 kB 00:00 2024-07-26T23:26:17.659 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:17.664 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:17.665 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:17.731 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:17.731 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:17.876 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:17.926 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-mirror-19.0.0-5372.g67f28c2a.el9.x86_64.rpm 8.9 MB/s | 3.2 MB 00:00 2024-07-26T23:26:17.927 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:17.927 INFO:teuthology.orchestra.run.smithi139.stdout:Total 8.8 MB/s | 3.2 MB 00:00 2024-07-26T23:26:17.927 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:17.936 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:17.936 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:17.981 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:18.016 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:18.016 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:18.446 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:18.464 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:18.660 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:18.686 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:18.686 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:26:18.687 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T23:26:18.687 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T23:26:18.687 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T23:26:18.687 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:18.690 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:18.690 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:18.690 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:18.690 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:18.691 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:18.691 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:18.961 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2024-07-26T23:26:19.523 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:54 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:19.612 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:19.641 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:19.641 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:19.657 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:19.658 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:26:19.658 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:19.719 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2024-07-26T23:26:19.887 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-nbd 2024-07-26T23:26:20.265 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:55 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:20.396 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:20.396 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 32 k 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 32 k 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 142 k 2024-07-26T23:26:20.397 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:20.448 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:53 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:20.558 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-19.0.0-5372.g67f28c2a.el9.x86_6 198 kB/s | 32 kB 00:00 2024-07-26T23:26:20.558 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:20.559 INFO:teuthology.orchestra.run.smithi177.stdout:Total 196 kB/s | 32 kB 00:00 2024-07-26T23:26:20.559 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:20.562 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:20.562 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:20.578 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:20.579 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:20.579 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:20.580 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:20.580 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-07-26T23:26:20.580 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:20.580 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:20.580 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 170 k 2024-07-26T23:26:20.580 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:20.581 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:20.581 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:20.581 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:26:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:20.581 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 170 k 2024-07-26T23:26:20.581 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 490 k 2024-07-26T23:26:20.582 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:20.659 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:20.763 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:20.806 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-nbd-19.0.0-5372.g67f28c2a.el9.x86_64.rpm 755 kB/s | 170 kB 00:00 2024-07-26T23:26:20.807 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:20.807 INFO:teuthology.orchestra.run.smithi139.stdout:Total 751 kB/s | 170 kB 00:00 2024-07-26T23:26:20.807 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:20.815 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:20.815 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:20.877 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:20.878 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:21.026 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:21.112 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:21.173 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:21.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:21.415 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:21.415 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:21.415 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:21.415 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:21.415 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:21.576 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:21.591 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2024-07-26T23:26:21.841 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:21.841 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:21.841 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:21.841 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:21.841 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:21.841 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:22.093 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base-debuginfo 2024-07-26T23:26:22.139 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:57 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:22.225 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:22.268 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:22.269 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:26:22.269 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:22.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2024-07-26T23:26:22.657 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:55 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:22.790 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 70 M 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 25 M 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:22.791 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:22.792 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:22.792 INFO:teuthology.orchestra.run.smithi139.stdout:Install 2 Packages 2024-07-26T23:26:22.792 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:22.792 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 95 M 2024-07-26T23:26:22.792 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 586 M 2024-07-26T23:26:22.792 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:22.888 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:57 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:22.974 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:23.017 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:23.018 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:26:23.018 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:23.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2024-07-26T23:26:23.624 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:58 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:23.710 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:23.753 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:23.754 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:26:23.755 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:23.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2024-07-26T23:26:24.366 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:59 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:24.452 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:24.495 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:24.495 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:26:24.496 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:24.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2024-07-26T23:26:25.106 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:00 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:25.192 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:25.234 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:25.235 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:26:25.235 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:25.294 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2024-07-26T23:26:25.877 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:00 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:25.968 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:26:26.012 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:26.013 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:26:26.014 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:26.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2024-07-26T23:26:26.263 INFO:teuthology.orchestra.run.smithi139.stdout:(1/2): ceph-base-debuginfo-19.0.0-5372.g67f28c2 20 MB/s | 70 MB 00:03 2024-07-26T23:26:26.648 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:01 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:26.746 INFO:teuthology.orchestra.run.smithi139.stdout:(2/2): ceph-debuginfo-19.0.0-5372.g67f28c2a.el9 6.4 MB/s | 25 MB 00:03 2024-07-26T23:26:26.748 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:26.748 INFO:teuthology.orchestra.run.smithi139.stdout:Total 24 MB/s | 95 MB 00:03 2024-07-26T23:26:26.749 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:26:26.754 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:26:26.755 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:26:26.776 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:26.776 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 85 k 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:26:26.777 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:26.778 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 85 k 2024-07-26T23:26:26.778 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 231 k 2024-07-26T23:26:26.778 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:27.021 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-19.0.0-5372.g67f28c2a.el9.x86_64.rpm 352 kB/s | 85 kB 00:00 2024-07-26T23:26:27.022 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:27.022 INFO:teuthology.orchestra.run.smithi177.stdout:Total 350 kB/s | 85 kB 00:00 2024-07-26T23:26:27.022 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:27.030 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:27.030 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:27.097 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:27.097 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:27.264 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:27.365 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:26:27.366 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:26:27.369 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:27.781 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:28.047 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:28.047 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:28.047 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:28.047 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:28.048 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:28.048 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:28.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-07-26T23:26:28.815 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:03 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:28.946 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:28.947 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:26:28.947 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:28.947 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 3.2 M 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.2 M 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 14 M 2024-07-26T23:26:28.948 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:29.318 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-19.0.0-5372.g67f28c2a.el9.x86_64.rpm 8.6 MB/s | 3.2 MB 00:00 2024-07-26T23:26:29.318 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:29.318 INFO:teuthology.orchestra.run.smithi177.stdout:Total 8.6 MB/s | 3.2 MB 00:00 2024-07-26T23:26:29.318 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:29.406 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:29.407 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:29.848 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:29.909 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:29.935 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:29.935 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T23:26:29.935 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T23:26:29.935 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T23:26:29.935 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T23:26:29.935 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:31.030 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:31.030 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:31.031 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:31.031 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:31.031 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:31.031 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:31.208 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:26:31.266 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2024-07-26T23:26:31.834 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:06 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:31.968 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:31.969 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:31.969 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-07-26T23:26:31.969 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 170 k 2024-07-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:31.971 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 170 k 2024-07-26T23:26:31.971 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 490 k 2024-07-26T23:26:31.971 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:32.187 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-19.0.0-5372.g67f28c2a.el9.x86_64.rpm 781 kB/s | 170 kB 00:00 2024-07-26T23:26:32.188 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:32.188 INFO:teuthology.orchestra.run.smithi177.stdout:Total 775 kB/s | 170 kB 00:00 2024-07-26T23:26:32.188 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:32.195 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:32.196 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:32.258 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:32.258 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:32.535 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:32.963 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:33.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:26:33.204 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:33.204 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:33.204 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:33.204 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:33.204 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:33.407 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base-debuginfo 2024-07-26T23:26:33.965 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:08 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:34.097 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:34.097 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:34.097 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:26:34.097 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:34.097 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:34.097 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 70 M 2024-07-26T23:26:34.098 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-07-26T23:26:34.098 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 25 M 2024-07-26T23:26:34.098 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:34.098 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:34.098 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:34.098 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-07-26T23:26:34.098 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:34.099 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 95 M 2024-07-26T23:26:34.099 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 586 M 2024-07-26T23:26:34.099 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:26:35.753 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): ceph-debuginfo-19.0.0-5372.g67f28c2a.el9 15 MB/s | 25 MB 00:01 2024-07-26T23:26:36.711 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-base-debuginfo-19.0.0-5372.g67f28c2 27 MB/s | 70 MB 00:02 2024-07-26T23:26:36.713 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:26:36.713 INFO:teuthology.orchestra.run.smithi177.stdout:Total 36 MB/s | 95 MB 00:02 2024-07-26T23:26:36.714 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:26:36.718 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:26:36.718 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:26:37.321 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:26:37.322 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:26:38.461 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/2 2024-07-26T23:26:38.546 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 2/2 2024-07-26T23:26:41.185 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:26:42.254 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 2/2 2024-07-26T23:26:42.254 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/2 2024-07-26T23:26:42.495 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/2 2024-07-26T23:26:42.495 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:42.495 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:26:42.495 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:42.495 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:42.495 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:42.496 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:26:42.738 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-common-debuginfo 2024-07-26T23:26:43.301 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:16 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:26:43.440 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:26:43.441 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:43.441 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:26:43.441 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:43.441 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 464 M 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 464 M 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 2.0 G 2024-07-26T23:26:43.442 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:26:48.427 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/2 2024-07-26T23:26:48.506 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 2/2 2024-07-26T23:26:52.329 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 2/2 2024-07-26T23:26:52.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/2 2024-07-26T23:26:52.559 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2/2 2024-07-26T23:26:52.559 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:52.559 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:26:52.559 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:52.559 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:26:52.559 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:52.559 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:26:52.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-common-debuginfo 2024-07-26T23:26:53.342 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:28 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:26:53.470 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:26:53.470 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:53.471 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:26:53.471 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:53.471 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:26:53.471 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 464 M 2024-07-26T23:26:53.471 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:53.472 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:26:53.472 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:26:53.472 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:26:53.472 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:26:53.472 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 464 M 2024-07-26T23:26:53.472 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.0 G 2024-07-26T23:26:53.472 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:27:02.908 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-common-debuginfo-19.0.0-5372.g67f28c2a.el9 24 MB/s | 464 MB 00:19 2024-07-26T23:27:02.908 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:27:02.909 INFO:teuthology.orchestra.run.smithi139.stdout:Total 24 MB/s | 464 MB 00:19 2024-07-26T23:27:02.909 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:27:02.914 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:27:02.915 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:27:05.661 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:27:05.662 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:27:08.134 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-common-debuginfo-19.0.0-5372.g67f28c2a.el9 32 MB/s | 464 MB 00:14 2024-07-26T23:27:08.135 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:27:08.136 INFO:teuthology.orchestra.run.smithi177.stdout:Total 32 MB/s | 464 MB 00:14 2024-07-26T23:27:08.136 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:27:08.140 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:27:08.140 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:27:10.879 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:27:10.880 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:27:53.276 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:27:53.383 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:27:58.548 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:27:58.653 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:28:01.506 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:28:01.732 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:28:01.732 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:01.732 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:28:01.732 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:28:01.732 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:01.732 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:28:01.998 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-07-26T23:28:02.554 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:03:35 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:28:02.683 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:28:02.683 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================== 2024-07-26T23:28:02.683 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================== 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.5 M 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================== 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:28:02.684 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:02.685 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 1.5 M 2024-07-26T23:28:02.685 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 6.7 M 2024-07-26T23:28:02.685 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:28:02.840 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:28:03.012 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-53 4.5 MB/s | 1.5 MB 00:00 2024-07-26T23:28:03.013 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:28:03.013 INFO:teuthology.orchestra.run.smithi139.stdout:Total 4.5 MB/s | 1.5 MB 00:00 2024-07-26T23:28:03.013 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:28:03.017 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:28:03.017 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:28:03.059 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:28:03.059 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:28:03.081 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:28:03.081 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:03.081 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:28:03.081 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:28:03.081 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:03.082 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:28:03.321 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:28:03.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-07-26T23:28:03.410 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-5372. 1/1 2024-07-26T23:28:03.760 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-5372. 1/1 2024-07-26T23:28:03.944 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:38 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:28:03.993 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-5372. 1/1 2024-07-26T23:28:03.993 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:03.993 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:28:03.993 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:28:03.993 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:03.993 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:28:04.084 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:28:04.085 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-07-26T23:28:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:28:04.085 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-07-26T23:28:04.085 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:28:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.5 M 2024-07-26T23:28:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:04.085 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:28:04.086 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-07-26T23:28:04.086 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:28:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:04.086 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.5 M 2024-07-26T23:28:04.086 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 6.7 M 2024-07-26T23:28:04.086 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:28:04.158 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw-debuginfo 2024-07-26T23:28:04.399 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-53 4.7 MB/s | 1.5 MB 00:00 2024-07-26T23:28:04.399 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:28:04.400 INFO:teuthology.orchestra.run.smithi177.stdout:Total 4.7 MB/s | 1.5 MB 00:00 2024-07-26T23:28:04.400 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:28:04.404 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:28:04.404 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:28:04.446 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:28:04.447 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:28:04.706 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:03:37 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:28:04.707 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:28:04.790 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-5372. 1/1 2024-07-26T23:28:04.835 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 201 M 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:28:04.836 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:28:04.837 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:04.837 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 201 M 2024-07-26T23:28:04.837 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 989 M 2024-07-26T23:28:04.837 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:28:05.227 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-5372. 1/1 2024-07-26T23:28:05.454 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-5372. 1/1 2024-07-26T23:28:05.454 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:05.454 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:28:05.454 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:28:05.454 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:05.454 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:28:05.622 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw-debuginfo 2024-07-26T23:28:06.168 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:41 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:28:06.297 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:28:06.297 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:28:06.297 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:28:06.297 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 201 M 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 201 M 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 989 M 2024-07-26T23:28:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:28:14.206 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-radosgw-debuginfo-19.0.0-5372.g67f28c2a.el 21 MB/s | 201 MB 00:09 2024-07-26T23:28:14.207 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:28:14.207 INFO:teuthology.orchestra.run.smithi139.stdout:Total 21 MB/s | 201 MB 00:09 2024-07-26T23:28:14.208 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:28:14.211 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:28:14.211 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:28:14.884 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-radosgw-debuginfo-19.0.0-5372.g67f28c2a.el 23 MB/s | 201 MB 00:08 2024-07-26T23:28:14.885 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:28:14.885 INFO:teuthology.orchestra.run.smithi177.stdout:Total 23 MB/s | 201 MB 00:08 2024-07-26T23:28:14.886 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:28:14.890 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:28:14.890 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:28:15.417 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:28:15.417 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:28:16.099 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:28:16.099 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:28:37.354 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:28:37.451 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x 1/1 2024-07-26T23:28:37.594 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:28:37.703 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x 1/1 2024-07-26T23:28:43.396 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x 1/1 2024-07-26T23:28:43.577 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x 1/1 2024-07-26T23:28:43.624 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x 1/1 2024-07-26T23:28:43.624 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:43.624 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:28:43.624 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:28:43.624 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:43.624 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:28:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x 1/1 2024-07-26T23:28:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:43.822 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:28:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:28:43.822 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:43.822 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:28:43.838 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-test-debuginfo 2024-07-26T23:28:44.037 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test-debuginfo 2024-07-26T23:28:44.386 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:04:17 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:28:44.515 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:28:44.515 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:28:44.515 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:28:44.515 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:28:44.515 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:28:44.515 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 721 M 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 721 M 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 3.6 G 2024-07-26T23:28:44.516 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:28:44.589 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:19 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:28:44.718 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:28:44.718 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:28:44.718 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 721 M 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 721 M 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 3.6 G 2024-07-26T23:28:44.719 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:29:14.927 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-test-debuginfo-19.0.0-5372.g67f28c2a.el9.x 24 MB/s | 721 MB 00:30 2024-07-26T23:29:14.928 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:29:14.928 INFO:teuthology.orchestra.run.smithi139.stdout:Total 24 MB/s | 721 MB 00:30 2024-07-26T23:29:14.929 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:29:14.934 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:29:14.934 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:29:19.181 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:29:19.181 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:29:25.511 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-test-debuginfo-19.0.0-5372.g67f28c2a.el9.x 18 MB/s | 721 MB 00:40 2024-07-26T23:29:25.512 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:29:25.512 INFO:teuthology.orchestra.run.smithi177.stdout:Total 18 MB/s | 721 MB 00:40 2024-07-26T23:29:25.512 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:29:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:29:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:29:29.769 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:29:29.769 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:30:37.322 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:30:37.442 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:30:47.105 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:30:47.351 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:30:47.351 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:47.351 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:30:47.352 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:30:47.352 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:47.352 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:30:47.656 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base-debuginfo 2024-07-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:30:48.078 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:30:48.203 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:21 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:30:48.290 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:30:48.333 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:30:48.334 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-07-26T23:30:48.334 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:30:48.414 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-debuginfo 2024-07-26T23:30:48.959 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:21 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:30:49.088 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 21 M 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:30:49.089 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:49.090 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 21 M 2024-07-26T23:30:49.090 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 106 M 2024-07-26T23:30:49.090 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:30:49.896 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-mgr-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 25 MB/s | 21 MB 00:00 2024-07-26T23:30:49.896 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:30:49.897 INFO:teuthology.orchestra.run.smithi139.stdout:Total 25 MB/s | 21 MB 00:00 2024-07-26T23:30:49.897 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:30:49.901 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:30:49.901 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:30:50.057 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:30:50.057 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:30:52.365 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:30:52.476 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:30:53.425 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:30:53.651 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:30:53.651 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:53.651 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:30:53.651 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:30:53.651 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:53.652 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:30:53.841 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mds-debuginfo 2024-07-26T23:30:54.384 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:27 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:30:54.512 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 53 M 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:30:54.513 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:30:54.514 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:30:54.514 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 53 M 2024-07-26T23:30:54.514 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 192 M 2024-07-26T23:30:54.514 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:30:56.131 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-mds-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 33 MB/s | 53 MB 00:01 2024-07-26T23:30:56.132 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:30:56.132 INFO:teuthology.orchestra.run.smithi139.stdout:Total 33 MB/s | 53 MB 00:01 2024-07-26T23:30:56.133 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:30:56.137 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:30:56.137 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:30:56.481 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:30:56.481 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:30:57.779 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:30:57.998 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:30:57.998 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:30:57.998 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:30:57.998 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:30:57.998 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:30:57.998 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:30:58.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base-debuginfo 2024-07-26T23:30:58.822 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:33 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:30:58.908 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 is already installed. 2024-07-26T23:30:58.950 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:30:58.951 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-07-26T23:30:58.951 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:30:59.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-debuginfo 2024-07-26T23:30:59.553 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:34 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 21 M 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:30:59.685 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:30:59.686 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:30:59.686 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:30:59.686 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:30:59.686 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 21 M 2024-07-26T23:30:59.686 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 106 M 2024-07-26T23:30:59.686 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:31:00.464 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mgr-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 26 MB/s | 21 MB 00:00 2024-07-26T23:31:00.465 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:31:00.466 INFO:teuthology.orchestra.run.smithi177.stdout:Total 26 MB/s | 21 MB 00:00 2024-07-26T23:31:00.466 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:31:00.469 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:31:00.469 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:31:00.624 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:31:00.624 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:31:01.721 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:31:01.828 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:31:03.029 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:03.293 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:03.524 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:03.525 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:31:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:31:03.525 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:03.525 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:31:03.713 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mon-debuginfo 2024-07-26T23:31:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:04.254 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:04.254 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:04.254 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:31:04.254 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:31:04.254 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:04.255 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:31:04.261 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:37 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:31:04.389 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:31:04.390 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:31:04.390 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:31:04.390 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:31:04.390 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:31:04.390 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 100 M 2024-07-26T23:31:04.390 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:04.390 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:31:04.391 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:31:04.391 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:31:04.391 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:04.391 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 100 M 2024-07-26T23:31:04.391 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 451 M 2024-07-26T23:31:04.391 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:31:04.436 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mds-debuginfo 2024-07-26T23:31:04.985 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:39 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:31:05.114 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 53 M 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:05.115 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:31:05.116 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:05.116 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 53 M 2024-07-26T23:31:05.116 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 192 M 2024-07-26T23:31:05.116 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:31:07.274 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-mon-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 35 MB/s | 100 MB 00:02 2024-07-26T23:31:07.275 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:31:07.276 INFO:teuthology.orchestra.run.smithi139.stdout:Total 35 MB/s | 100 MB 00:02 2024-07-26T23:31:07.276 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:31:07.280 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:31:07.280 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:31:07.904 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:31:07.904 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:31:07.992 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mds-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 18 MB/s | 53 MB 00:02 2024-07-26T23:31:07.993 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:31:07.993 INFO:teuthology.orchestra.run.smithi177.stdout:Total 18 MB/s | 53 MB 00:02 2024-07-26T23:31:07.994 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:31:07.998 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:31:07.998 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:31:08.342 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:31:08.342 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:31:13.574 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:31:13.672 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:15.086 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:15.311 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:15.311 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:15.311 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:31:15.311 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:31:15.312 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:15.312 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:31:15.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mon-debuginfo 2024-07-26T23:31:16.027 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:06:51 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:31:16.156 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:31:16.156 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 100 M 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 100 M 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 451 M 2024-07-26T23:31:16.157 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:31:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:31:18.397 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:19.010 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-mon-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 35 MB/s | 100 MB 00:02 2024-07-26T23:31:19.011 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:31:19.011 INFO:teuthology.orchestra.run.smithi177.stdout:Total 35 MB/s | 100 MB 00:02 2024-07-26T23:31:19.012 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:31:19.016 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:31:19.016 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:31:19.636 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:31:19.637 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:31:21.270 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:21.521 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:21.521 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:31:21.521 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:31:21.521 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:21.521 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:31:21.730 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-osd-debuginfo 2024-07-26T23:31:22.275 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:55 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:31:22.405 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 290 M 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:31:22.406 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 290 M 2024-07-26T23:31:22.407 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 1.2 G 2024-07-26T23:31:22.407 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:31:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:31:30.149 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:32.993 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:33.257 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:31:33.257 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:33.257 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:31:33.257 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:31:33.257 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:33.257 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:31:33.399 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-osd-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 26 MB/s | 290 MB 00:10 2024-07-26T23:31:33.400 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:31:33.401 INFO:teuthology.orchestra.run.smithi139.stdout:Total 26 MB/s | 290 MB 00:10 2024-07-26T23:31:33.401 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:31:33.405 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:31:33.406 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:31:33.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-osd-debuginfo 2024-07-26T23:31:33.997 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:08 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:31:34.126 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:31:34.127 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:34.127 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:31:34.127 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:34.127 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 290 M 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 290 M 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.2 G 2024-07-26T23:31:34.128 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:31:35.128 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:31:35.129 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:31:43.194 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-osd-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 32 MB/s | 290 MB 00:09 2024-07-26T23:31:43.195 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:31:43.195 INFO:teuthology.orchestra.run.smithi177.stdout:Total 32 MB/s | 290 MB 00:09 2024-07-26T23:31:43.195 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:31:43.199 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:31:43.200 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:31:44.925 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:31:44.926 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:04.677 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:04.785 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:12.111 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:12.345 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:12.345 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:12.345 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:12.345 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:12.345 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:12.346 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:12.567 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-fuse-debuginfo 2024-07-26T23:32:13.114 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:07:46 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:13.243 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 6.1 M 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:13.244 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 6.1 M 2024-07-26T23:32:13.245 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 29 M 2024-07-26T23:32:13.245 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:13.689 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-fuse-debuginfo-19.0.0-5372.g67f28c2a.el9.x 14 MB/s | 6.1 MB 00:00 2024-07-26T23:32:13.689 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:13.690 INFO:teuthology.orchestra.run.smithi139.stdout:Total 14 MB/s | 6.1 MB 00:00 2024-07-26T23:32:13.690 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:13.694 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:13.694 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:13.764 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:13.765 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:14.522 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:14.541 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:14.629 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:15.126 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:15.370 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:15.370 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:15.370 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:15.370 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:15.370 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:15.371 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:15.564 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados-devel-debuginfo 2024-07-26T23:32:16.107 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:07:49 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:16.235 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:16.236 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:16.236 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:16.236 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:16.236 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:16.236 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 108 k 2024-07-26T23:32:16.236 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:16.237 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:16.237 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:16.237 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:16.237 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:16.237 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 108 k 2024-07-26T23:32:16.237 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 437 k 2024-07-26T23:32:16.237 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:16.458 INFO:teuthology.orchestra.run.smithi139.stdout:librados-devel-debuginfo-19.0.0-5372.g67f28c2a. 487 kB/s | 108 kB 00:00 2024-07-26T23:32:16.459 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:16.459 INFO:teuthology.orchestra.run.smithi139.stdout:Total 483 kB/s | 108 kB 00:00 2024-07-26T23:32:16.459 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:16.462 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:16.463 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:16.498 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:16.498 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:16.616 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:16.724 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:17.100 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:17.335 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:17.335 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:17.335 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:17.336 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:17.336 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:17.336 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:17.522 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs2-debuginfo 2024-07-26T23:32:18.072 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:07:51 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:18.201 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:18.202 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:18.202 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:18.202 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:18.202 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:18.202 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 5.5 M 2024-07-26T23:32:18.202 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:18.203 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:18.203 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:18.203 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:18.203 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:18.203 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 5.5 M 2024-07-26T23:32:18.203 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 25 M 2024-07-26T23:32:18.203 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:18.619 INFO:teuthology.orchestra.run.smithi139.stdout:libcephfs2-debuginfo-19.0.0-5372.g67f28c2a.el9. 13 MB/s | 5.5 MB 00:00 2024-07-26T23:32:18.620 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:18.620 INFO:teuthology.orchestra.run.smithi139.stdout:Total 13 MB/s | 5.5 MB 00:00 2024-07-26T23:32:18.620 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:18.624 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:18.624 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:18.690 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:18.690 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:19.366 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:19.458 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:32:19.947 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:32:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:32:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:20.193 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:20.193 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:20.408 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados2-debuginfo 2024-07-26T23:32:20.954 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:07:53 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:21.084 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:21.084 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:21.084 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:21.084 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:21.084 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 46 M 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 46 M 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 256 M 2024-07-26T23:32:21.085 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:22.147 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:22.384 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:22.384 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:22.384 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:22.384 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:22.384 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:22.385 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:22.572 INFO:teuthology.orchestra.run.smithi139.stdout:librados2-debuginfo-19.0.0-5372.g67f28c2a.el9.x 31 MB/s | 46 MB 00:01 2024-07-26T23:32:22.573 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:22.574 INFO:teuthology.orchestra.run.smithi139.stdout:Total 31 MB/s | 46 MB 00:01 2024-07-26T23:32:22.574 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:22.578 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:22.578 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:22.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse-debuginfo 2024-07-26T23:32:22.885 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:22.885 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:23.147 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:07:58 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:23.276 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 6.1 M 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:23.277 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:23.278 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:23.278 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 6.1 M 2024-07-26T23:32:23.278 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 29 M 2024-07-26T23:32:23.278 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:23.709 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-fuse-debuginfo-19.0.0-5372.g67f28c2a.el9.x 14 MB/s | 6.1 MB 00:00 2024-07-26T23:32:23.709 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:23.710 INFO:teuthology.orchestra.run.smithi177.stdout:Total 14 MB/s | 6.1 MB 00:00 2024-07-26T23:32:23.710 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:23.713 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:23.713 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:23.784 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:23.784 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:24.551 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:24.647 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:25.172 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:25.417 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:25.417 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:25.418 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:25.418 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:25.418 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:25.418 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:25.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel-debuginfo 2024-07-26T23:32:26.146 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:01 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:26.275 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:26.276 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:26.276 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:26.276 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:26.276 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:26.276 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 108 k 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 108 k 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 437 k 2024-07-26T23:32:26.277 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:26.487 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-debuginfo-19.0.0-5372.g67f28c2a. 514 kB/s | 108 kB 00:00 2024-07-26T23:32:26.487 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:26.487 INFO:teuthology.orchestra.run.smithi177.stdout:Total 511 kB/s | 108 kB 00:00 2024-07-26T23:32:26.488 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:26.491 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:26.491 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:26.525 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:26.526 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:26.639 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:26.751 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:27.061 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:27.309 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:27.309 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:27.309 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:27.309 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:27.309 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:27.309 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:27.500 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2-debuginfo 2024-07-26T23:32:28.043 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:28.048 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:03 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:28.137 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:28.179 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:28.180 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:28.180 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:28.180 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 5.5 M 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:28.181 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 5.5 M 2024-07-26T23:32:28.182 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 25 M 2024-07-26T23:32:28.182 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:28.606 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs2-debuginfo-19.0.0-5372.g67f28c2a.el9. 13 MB/s | 5.5 MB 00:00 2024-07-26T23:32:28.607 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:28.607 INFO:teuthology.orchestra.run.smithi177.stdout:Total 13 MB/s | 5.5 MB 00:00 2024-07-26T23:32:28.608 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:28.611 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:28.611 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:28.677 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:28.677 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:29.384 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:29.471 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:32:29.931 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:32:29.958 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:30.174 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:32:30.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:30.175 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:30.175 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:30.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:30.175 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:30.184 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:30.184 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:30.184 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:30.184 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:30.184 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:30.184 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:30.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2-debuginfo 2024-07-26T23:32:30.385 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librbd1-debuginfo 2024-07-26T23:32:30.922 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:05 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:30.932 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:03 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:31.052 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 46 M 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:31.053 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:31.054 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:31.054 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:31.054 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 46 M 2024-07-26T23:32:31.054 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 256 M 2024-07-26T23:32:31.054 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:31.062 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:31.062 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:31.062 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:31.062 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:31.062 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:31.063 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 53 M 2024-07-26T23:32:31.063 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:31.063 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:31.063 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:31.063 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:31.063 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:31.063 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 53 M 2024-07-26T23:32:31.064 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 312 M 2024-07-26T23:32:31.064 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:33.043 INFO:teuthology.orchestra.run.smithi177.stdout:librados2-debuginfo-19.0.0-5372.g67f28c2a.el9.x 23 MB/s | 46 MB 00:01 2024-07-26T23:32:33.043 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:33.044 INFO:teuthology.orchestra.run.smithi177.stdout:Total 23 MB/s | 46 MB 00:01 2024-07-26T23:32:33.044 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:33.048 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:33.048 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:33.354 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:33.354 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:33.433 INFO:teuthology.orchestra.run.smithi139.stdout:librbd1-debuginfo-19.0.0-5372.g67f28c2a.el9.x86 22 MB/s | 53 MB 00:02 2024-07-26T23:32:33.433 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:33.434 INFO:teuthology.orchestra.run.smithi139.stdout:Total 22 MB/s | 53 MB 00:02 2024-07-26T23:32:33.434 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:33.438 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:33.438 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:33.787 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:33.788 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:38.497 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:38.598 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:39.861 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:39.964 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:32:40.411 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:40.638 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_ 1/1 2024-07-26T23:32:40.638 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:40.638 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:40.638 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:40.638 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:40.638 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:40.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1-debuginfo 2024-07-26T23:32:41.417 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:16 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:41.545 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 53 M 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:41.546 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:41.547 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:41.547 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:41.547 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 53 M 2024-07-26T23:32:41.547 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 312 M 2024-07-26T23:32:41.547 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:42.127 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:32:42.373 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:32:42.373 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:42.373 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:42.373 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:42.373 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:42.373 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:42.578 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-cephfs-debuginfo 2024-07-26T23:32:43.124 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:16 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:43.187 INFO:teuthology.orchestra.run.smithi177.stdout:librbd1-debuginfo-19.0.0-5372.g67f28c2a.el9.x86 32 MB/s | 53 MB 00:01 2024-07-26T23:32:43.187 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:43.188 INFO:teuthology.orchestra.run.smithi177.stdout:Total 32 MB/s | 53 MB 00:01 2024-07-26T23:32:43.188 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:43.192 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:43.192 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:43.253 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 721 k 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:43.254 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 721 k 2024-07-26T23:32:43.255 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 2.4 M 2024-07-26T23:32:43.255 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:43.542 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:43.542 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:43.546 INFO:teuthology.orchestra.run.smithi139.stdout:python3-cephfs-debuginfo-19.0.0-5372.g67f28c2a. 2.4 MB/s | 721 kB 00:00 2024-07-26T23:32:43.547 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:43.547 INFO:teuthology.orchestra.run.smithi139.stdout:Total 2.4 MB/s | 721 kB 00:00 2024-07-26T23:32:43.547 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:43.550 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:43.551 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:43.589 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:43.589 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:43.764 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:43.857 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:44.224 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:44.458 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:44.458 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:44.458 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:44.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:44.458 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:44.458 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:44.664 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rados-debuginfo 2024-07-26T23:32:45.213 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:18 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:45.344 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:45.344 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:45.344 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:45.344 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:45.344 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:45.344 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 892 k 2024-07-26T23:32:45.344 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:45.345 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:45.345 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:45.345 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:45.345 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:45.345 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 892 k 2024-07-26T23:32:45.345 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 3.1 M 2024-07-26T23:32:45.345 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:45.621 INFO:teuthology.orchestra.run.smithi139.stdout:python3-rados-debuginfo-19.0.0-5372.g67f28c2a.e 3.2 MB/s | 892 kB 00:00 2024-07-26T23:32:45.622 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:45.622 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.1 MB/s | 892 kB 00:00 2024-07-26T23:32:45.622 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:45.625 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:45.626 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:45.665 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:45.666 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:45.855 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:45.952 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9. 1/1 2024-07-26T23:32:46.344 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9. 1/1 2024-07-26T23:32:46.563 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9. 1/1 2024-07-26T23:32:46.563 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:46.563 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:46.563 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:46.563 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:46.563 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:46.769 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rbd-debuginfo 2024-07-26T23:32:47.318 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:20 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:47.446 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:47.447 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:47.447 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:47.447 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:47.447 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:47.447 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.4 M 2024-07-26T23:32:47.447 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:47.448 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:47.448 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:47.448 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:47.448 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:47.448 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 1.4 M 2024-07-26T23:32:47.448 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 5.1 M 2024-07-26T23:32:47.448 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:47.757 INFO:teuthology.orchestra.run.smithi139.stdout:python3-rbd-debuginfo-19.0.0-5372.g67f28c2a.el9 4.4 MB/s | 1.4 MB 00:00 2024-07-26T23:32:47.757 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:47.758 INFO:teuthology.orchestra.run.smithi139.stdout:Total 4.4 MB/s | 1.4 MB 00:00 2024-07-26T23:32:47.758 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:47.761 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:47.761 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:47.804 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:47.805 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:48.047 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:48.514 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:48.748 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:48.748 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:48.748 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:48.748 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:48.748 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:48.748 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:48.967 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rgw-debuginfo 2024-07-26T23:32:49.515 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:22 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:49.632 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:49.644 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 394 k 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:49.646 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:49.647 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:49.647 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 394 k 2024-07-26T23:32:49.647 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 1.2 M 2024-07-26T23:32:49.647 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:49.742 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:32:49.927 INFO:teuthology.orchestra.run.smithi139.stdout:python3-rgw-debuginfo-19.0.0-5372.g67f28c2a.el9 1.4 MB/s | 394 kB 00:00 2024-07-26T23:32:49.928 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:49.928 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.4 MB/s | 394 kB 00:00 2024-07-26T23:32:49.928 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:49.932 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:49.932 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:49.969 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:49.969 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:50.113 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:50.247 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:50.629 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:50.872 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:50.872 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:50.872 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:50.872 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:50.872 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:50.872 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:51.054 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-fuse-debuginfo 2024-07-26T23:32:51.608 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:24 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:51.737 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:51.738 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:51.738 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:51.738 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:51.738 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:51.738 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 518 k 2024-07-26T23:32:51.738 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:51.739 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:51.739 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:51.739 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:51.739 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:51.739 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 518 k 2024-07-26T23:32:51.739 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 2.6 M 2024-07-26T23:32:51.739 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:51.939 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:32:52.013 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-fuse-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 1.8 MB/s | 518 kB 00:00 2024-07-26T23:32:52.014 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:52.014 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.8 MB/s | 518 kB 00:00 2024-07-26T23:32:52.014 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:52.017 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:52.017 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:52.055 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:52.055 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:52.171 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:32:52.171 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:52.171 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:52.171 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:52.172 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:52.172 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:52.216 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:32:52.329 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:52.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs-debuginfo 2024-07-26T23:32:52.703 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:52.920 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:27 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:32:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:52.930 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:32:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:52.931 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:32:53.050 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:53.051 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:53.051 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:53.051 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:53.051 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 721 k 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 721 k 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.4 M 2024-07-26T23:32:53.052 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:53.135 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-mirror-debuginfo 2024-07-26T23:32:53.339 INFO:teuthology.orchestra.run.smithi177.stdout:python3-cephfs-debuginfo-19.0.0-5372.g67f28c2a. 2.5 MB/s | 721 kB 00:00 2024-07-26T23:32:53.340 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:53.340 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.4 MB/s | 721 kB 00:00 2024-07-26T23:32:53.340 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:53.343 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:53.344 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:53.383 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:53.383 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:53.562 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:53.654 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:53.680 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:26 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:32:53.810 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:32:53.810 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 52 M 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:32:53.811 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:32:53.812 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 52 M 2024-07-26T23:32:53.812 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 301 M 2024-07-26T23:32:53.812 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:32:54.013 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:54.235 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9 1/1 2024-07-26T23:32:54.236 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:54.236 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:54.236 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:54.236 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:54.236 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:54.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados-debuginfo 2024-07-26T23:32:54.983 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:29 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:55.113 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 892 k 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:55.114 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:55.115 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 892 k 2024-07-26T23:32:55.115 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 3.1 M 2024-07-26T23:32:55.115 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:55.408 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-mirror-debuginfo-19.0.0-5372.g67f28c2a.el9. 33 MB/s | 52 MB 00:01 2024-07-26T23:32:55.409 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rados-debuginfo-19.0.0-5372.g67f28c2a.e 3.0 MB/s | 892 kB 00:00 2024-07-26T23:32:55.409 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:55.409 INFO:teuthology.orchestra.run.smithi139.stdout:Total 33 MB/s | 52 MB 00:01 2024-07-26T23:32:55.410 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:55.410 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.0 MB/s | 892 kB 00:00 2024-07-26T23:32:55.410 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:55.410 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:32:55.413 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:55.413 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:55.413 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:32:55.413 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:32:55.452 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:55.453 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:55.646 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:55.741 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9. 1/1 2024-07-26T23:32:55.755 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:32:55.756 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:32:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9. 1/1 2024-07-26T23:32:56.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9. 1/1 2024-07-26T23:32:56.353 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:56.353 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:56.353 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:56.354 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:56.354 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:56.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd-debuginfo 2024-07-26T23:32:57.107 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:32 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:57.236 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:57.236 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.4 M 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.4 M 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 5.1 M 2024-07-26T23:32:57.237 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:57.545 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rbd-debuginfo-19.0.0-5372.g67f28c2a.el9 4.4 MB/s | 1.4 MB 00:00 2024-07-26T23:32:57.546 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:57.546 INFO:teuthology.orchestra.run.smithi177.stdout:Total 4.4 MB/s | 1.4 MB 00:00 2024-07-26T23:32:57.546 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:57.549 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:57.549 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:57.592 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:57.592 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:57.837 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:57.936 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:58.321 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:58.552 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:32:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:58.553 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:32:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:32:58.553 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:58.553 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:32:58.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw-debuginfo 2024-07-26T23:32:59.292 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:34 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:32:59.422 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 394 k 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:32:59.423 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 394 k 2024-07-26T23:32:59.424 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.2 M 2024-07-26T23:32:59.424 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:32:59.699 INFO:teuthology.orchestra.run.smithi177.stdout:python3-rgw-debuginfo-19.0.0-5372.g67f28c2a.el9 1.4 MB/s | 394 kB 00:00 2024-07-26T23:32:59.699 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:32:59.700 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.4 MB/s | 394 kB 00:00 2024-07-26T23:32:59.700 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:32:59.703 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:32:59.703 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:32:59.740 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:32:59.740 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:32:59.879 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:32:59.979 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:33:00.357 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:33:00.603 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x8 1/1 2024-07-26T23:33:00.603 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:00.603 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:33:00.603 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:33:00.603 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:00.603 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:33:00.781 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse-debuginfo 2024-07-26T23:33:01.334 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:36 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:33:01.465 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:33:01.465 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 518 k 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 518 k 2024-07-26T23:33:01.466 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.6 M 2024-07-26T23:33:01.467 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:33:01.718 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:33:01.760 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-debuginfo-19.0.0-5372.g67f28c2a.el9.x8 1.7 MB/s | 518 kB 00:00 2024-07-26T23:33:01.760 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:33:01.760 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.7 MB/s | 518 kB 00:00 2024-07-26T23:33:01.761 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:33:01.764 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:33:01.764 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:33:01.803 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:33:01.803 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:33:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:33:01.964 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:33:02.058 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:33:02.443 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:33:02.686 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_6 1/1 2024-07-26T23:33:02.686 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:02.686 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:33:02.686 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:33:02.686 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:02.686 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:33:02.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror-debuginfo 2024-07-26T23:33:03.419 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:38 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:33:03.549 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:33:03.550 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 52 M 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 52 M 2024-07-26T23:33:03.551 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 301 M 2024-07-26T23:33:03.552 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:33:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:33:04.110 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:33:04.110 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:04.110 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:33:04.110 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:33:04.110 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:04.111 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:33:04.322 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-nbd-debuginfo 2024-07-26T23:33:04.864 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:08:37 ago on Fri 26 Jul 2024 11:24:27 PM UTC. 2024-07-26T23:33:04.994 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-07-26T23:33:04.994 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.3 M 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 1.3 M 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 5.8 M 2024-07-26T23:33:04.995 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-07-26T23:33:05.188 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-debuginfo-19.0.0-5372.g67f28c2a.el9. 32 MB/s | 52 MB 00:01 2024-07-26T23:33:05.189 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:33:05.189 INFO:teuthology.orchestra.run.smithi177.stdout:Total 32 MB/s | 52 MB 00:01 2024-07-26T23:33:05.189 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:33:05.193 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:33:05.193 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:33:05.326 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-nbd-debuginfo-19.0.0-5372.g67f28c2a.el9.x86 3.9 MB/s | 1.3 MB 00:00 2024-07-26T23:33:05.327 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:33:05.327 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.9 MB/s | 1.3 MB 00:00 2024-07-26T23:33:05.327 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-07-26T23:33:05.331 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-07-26T23:33:05.331 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-07-26T23:33:05.374 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-07-26T23:33:05.374 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-07-26T23:33:05.538 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:33:05.538 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:33:05.610 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-07-26T23:33:05.715 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:33:06.127 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:33:06.360 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:33:06.360 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:06.360 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-07-26T23:33:06.360 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:33:06.360 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:06.360 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-07-26T23:33:06.561 DEBUG:teuthology.parallel:result is None 2024-07-26T23:33:11.563 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:33:11.660 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:33:13.733 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:33:13.984 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86 1/1 2024-07-26T23:33:13.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:13.984 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:33:13.984 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:33:13.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:13.984 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:33:14.203 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd-debuginfo 2024-07-26T23:33:14.752 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:08:49 ago on Fri 26 Jul 2024 11:24:25 PM UTC. 2024-07-26T23:33:14.882 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-07-26T23:33:14.882 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-5372.g67f28c2a.el9 ceph 1.3 M 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 1.3 M 2024-07-26T23:33:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 5.8 M 2024-07-26T23:33:14.884 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-07-26T23:33:15.181 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-debuginfo-19.0.0-5372.g67f28c2a.el9.x86 4.3 MB/s | 1.3 MB 00:00 2024-07-26T23:33:15.182 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-07-26T23:33:15.182 INFO:teuthology.orchestra.run.smithi177.stdout:Total 4.3 MB/s | 1.3 MB 00:00 2024-07-26T23:33:15.182 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-07-26T23:33:15.186 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-07-26T23:33:15.186 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-07-26T23:33:15.228 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-07-26T23:33:15.228 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-07-26T23:33:15.465 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-07-26T23:33:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:33:15.932 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:33:16.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 1/1 2024-07-26T23:33:16.200 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:16.200 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-07-26T23:33:16.201 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-debuginfo-2:19.0.0-5372.g67f28c2a.el9.x86_64 2024-07-26T23:33:16.201 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:16.201 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-07-26T23:33:16.383 DEBUG:teuthology.parallel:result is None 2024-07-26T23:33:16.384 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=67f28c2a04ad1fe90ea612a39329cc6af2a92d8b 2024-07-26T23:33:16.535 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T23:33:16.560 INFO:teuthology.orchestra.run.smithi139.stdout:19.0.0-5372.g67f28c2a.el9 2024-07-26T23:33:16.561 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5372.g67f28c2a.el9 2024-07-26T23:33:16.561 INFO:teuthology.task.install:The correct ceph version 19.0.0-5372.g67f28c2a is installed. 2024-07-26T23:33:16.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=67f28c2a04ad1fe90ea612a39329cc6af2a92d8b 2024-07-26T23:33:16.703 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T23:33:16.724 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-5372.g67f28c2a.el9 2024-07-26T23:33:16.725 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5372.g67f28c2a.el9 2024-07-26T23:33:16.725 INFO:teuthology.task.install:The correct ceph version 19.0.0-5372.g67f28c2a is installed. 2024-07-26T23:33:16.727 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T23:33:16.727 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:16.727 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T23:33:16.757 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:16.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T23:33:16.791 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T23:33:16.791 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:16.791 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T23:33:16.826 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T23:33:16.894 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:16.894 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T23:33:16.919 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T23:33:16.983 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T23:33:16.983 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:16.983 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T23:33:17.010 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T23:33:17.075 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:17.075 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T23:33:17.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T23:33:17.163 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T23:33:17.164 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:17.164 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T23:33:17.189 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T23:33:17.254 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:17.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T23:33:17.279 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T23:33:17.344 INFO:teuthology.run_tasks:Running task ceph... 2024-07-26T23:33:17.428 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-07-26T23:33:17.429 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /var/log/ceph 2024-07-26T23:33:17.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-07-26T23:33:17.456 INFO:tasks.ceph:Disabling ceph logrotate... 2024-07-26T23:33:17.456 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/logrotate.d/ceph 2024-07-26T23:33:17.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-07-26T23:33:17.521 INFO:tasks.ceph:Creating extra log directories... 2024-07-26T23:33:17.521 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-07-26T23:33:17.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-07-26T23:33:17.639 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-07-26T23:33:17.639 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 160, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-07-26T23:33:17.639 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516', 'branch': 'wip-yuri-testing-2024-07-26-0628', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7820516', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2024-07-26-0628'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 160, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri-testing-2024-07-26-0628', 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 7667, 'sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri-testing-2024-07-26-0628', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b', 'targets': {'smithi139.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGh/OTKb9BauDrSJETdT5PDHA6QPNmwZ98jZ+aFieEWYLPX9+XzaVkG2ryVoOCIo8mbA0AaN+BaE6/DnOKL+UQA=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHJs+7Jlsb2eLt1qygUU+szB+erFTL0fwmvrqchqqzrptQDDElYXbPgt/hjVBrw/JSwqprjkotVIpANJtZE9ys8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '53ce1462e129f6eb4071986336534c740fdebd31', 'timestamp': '2024-07-26_21:23:43', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525'} 2024-07-26T23:33:17.640 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-07-26T23:33:17.642 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-07-26T23:33:17.657 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/run/ceph 2024-07-26T23:33:17.699 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-07-26T23:33:17.723 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:17.723 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T23:33:17.776 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T23:33:17.777 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2024-07-26T23:33:17.832 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T23:33:17.832 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:17.832 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-07-26T23:33:17.833 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:17.833 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:17.833 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-07-26 23:26:18.946362879 +0000 2024-07-26T23:33:17.833 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-07-26 23:23:15.037335258 +0000 2024-07-26T23:33:17.833 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-07-26 23:23:15.037335258 +0000 2024-07-26T23:33:17.833 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-07-26 23:23:15.037335258 +0000 2024-07-26T23:33:17.833 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T23:33:17.895 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-07-26T23:33:17.895 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-07-26T23:33:17.895 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000185679 s, 2.8 MB/s 2024-07-26T23:33:17.896 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T23:33:17.952 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 692 Links: 1 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-07-26 23:26:18.946362879 +0000 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-07-26 23:23:15.371328118 +0000 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-07-26 23:23:15.371328118 +0000 2024-07-26T23:33:18.009 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-07-26 23:23:15.371328118 +0000 2024-07-26T23:33:18.009 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T23:33:18.072 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-07-26T23:33:18.072 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-07-26T23:33:18.072 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000170414 s, 3.0 MB/s 2024-07-26T23:33:18.073 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T23:33:18.129 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2024-07-26T23:33:18.185 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T23:33:18.185 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:18.185 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 711 Links: 1 2024-07-26T23:33:18.185 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:18.185 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:18.186 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-07-26 23:26:18.946362879 +0000 2024-07-26T23:33:18.186 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-07-26 23:23:15.703321019 +0000 2024-07-26T23:33:18.186 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-07-26 23:23:15.703321019 +0000 2024-07-26T23:33:18.186 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-07-26 23:23:15.703321019 +0000 2024-07-26T23:33:18.186 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T23:33:18.248 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-07-26T23:33:18.249 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-07-26T23:33:18.249 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000206699 s, 2.5 MB/s 2024-07-26T23:33:18.250 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T23:33:18.309 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2024-07-26T23:33:18.365 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T23:33:18.365 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:18.365 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 733 Links: 1 2024-07-26T23:33:18.366 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:18.366 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:18.366 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-07-26 23:26:18.947362858 +0000 2024-07-26T23:33:18.366 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-07-26 23:23:16.035313922 +0000 2024-07-26T23:33:18.366 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-07-26 23:23:16.035313922 +0000 2024-07-26T23:33:18.366 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-07-26 23:23:16.035313922 +0000 2024-07-26T23:33:18.366 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T23:33:18.428 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-07-26T23:33:18.428 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-07-26T23:33:18.428 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000238765 s, 2.1 MB/s 2024-07-26T23:33:18.430 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T23:33:18.486 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:18.486 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:18.487 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T23:33:18.503 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T23:33:18.503 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-07-26 23:26:30.188267641 +0000 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-07-26 23:23:16.541057833 +0000 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-07-26 23:23:16.541057833 +0000 2024-07-26T23:33:18.560 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-07-26 23:23:16.541057833 +0000 2024-07-26T23:33:18.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T23:33:18.624 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-07-26T23:33:18.625 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-07-26T23:33:18.625 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000270255 s, 1.9 MB/s 2024-07-26T23:33:18.626 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T23:33:18.683 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-07-26T23:33:18.739 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T23:33:18.739 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:18.739 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-07-26T23:33:18.739 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:18.739 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:18.740 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-07-26 23:26:30.188267641 +0000 2024-07-26T23:33:18.740 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-07-26 23:23:16.887048476 +0000 2024-07-26T23:33:18.740 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-07-26 23:23:16.887048476 +0000 2024-07-26T23:33:18.740 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-07-26 23:23:16.887048476 +0000 2024-07-26T23:33:18.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T23:33:18.802 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-07-26T23:33:18.803 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-07-26T23:33:18.803 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.0001934 s, 2.6 MB/s 2024-07-26T23:33:18.804 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T23:33:18.861 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-07-26T23:33:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T23:33:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:18.917 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-07-26T23:33:18.917 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:18.918 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:18.918 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-07-26 23:26:30.188267641 +0000 2024-07-26T23:33:18.918 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-07-26 23:23:17.216039579 +0000 2024-07-26T23:33:18.918 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-07-26 23:23:17.216039579 +0000 2024-07-26T23:33:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-07-26 23:23:17.216039579 +0000 2024-07-26T23:33:18.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T23:33:18.980 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-07-26T23:33:18.980 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-07-26T23:33:18.980 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000184016 s, 2.8 MB/s 2024-07-26T23:33:18.981 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T23:33:19.037 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-07-26T23:33:19.092 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-07-26 23:26:30.189267614 +0000 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-07-26 23:23:17.558030331 +0000 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-07-26 23:23:17.558030331 +0000 2024-07-26T23:33:19.093 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-07-26 23:23:17.558030331 +0000 2024-07-26T23:33:19.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T23:33:19.156 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-07-26T23:33:19.157 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-07-26T23:33:19.157 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000188875 s, 2.7 MB/s 2024-07-26T23:33:19.158 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T23:33:19.214 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:19.215 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-07-26T23:33:19.215 INFO:tasks.ceph:Generating config... 2024-07-26T23:33:19.216 INFO:tasks.ceph:[client] debug monc = 20 2024-07-26T23:33:19.216 INFO:tasks.ceph:[client] debug ms = 1 2024-07-26T23:33:19.216 INFO:tasks.ceph:[client] debug objecter = 20 2024-07-26T23:33:19.216 INFO:tasks.ceph:[client] debug rados = 20 2024-07-26T23:33:19.216 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-07-26T23:33:19.216 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-07-26T23:33:19.216 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-07-26T23:33:19.217 INFO:tasks.ceph:[global] ms type = async 2024-07-26T23:33:19.217 INFO:tasks.ceph:[global] osd heartbeat grace = 160 2024-07-26T23:33:19.217 INFO:tasks.ceph:[global] osd op thread timeout = 150 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mgr] debug ms = 1 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] debug mon = 20 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] debug ms = 1 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] debug paxos = 20 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-07-26T23:33:19.217 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-07-26T23:33:19.218 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-07-26T23:33:19.218 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-07-26T23:33:19.218 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] bdev async discard = True 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] debug monc = 20 2024-07-26T23:33:19.218 INFO:tasks.ceph:[osd] debug ms = 1 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] debug osd = 20 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd class default list = * 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd class load list = * 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-07-26T23:33:19.219 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-07-26T23:33:19.220 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-07-26T23:33:19.220 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-07-26T23:33:19.220 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-07-26T23:33:19.220 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-07-26T23:33:19.220 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-07-26T23:33:19.220 INFO:tasks.ceph:Setting up mon.a... 2024-07-26T23:33:19.220 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-07-26T23:33:19.279 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/ceph.keyring 2024-07-26T23:33:19.282 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-07-26T23:33:19.326 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-26T23:33:19.392 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.139'), ('mon.c', '[v2:172.21.15.139:3301,v1:172.21.15.139:6790]'), ('mon.b', '172.21.15.177')] 2024-07-26T23:33:19.392 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.139,[v2:172.21.15.139:3301,v1:172.21.15.139:6790],172.21.15.177', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 160, 'osd op thread timeout': 150}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-07-26T23:33:19.393 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:19.393 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-07-26T23:33:19.448 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.139 --addv c '[v2:172.21.15.139:3301,v1:172.21.15.139:6790]' --add b 172.21.15.177 --print /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:19.536 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:19.536 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: generated fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:33:19.536 INFO:teuthology.orchestra.run.smithi139.stdout:setting min_mon_release = quincy 2024-07-26T23:33:19.536 INFO:teuthology.orchestra.run.smithi139.stdout:epoch 0 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:min_mon_release 17 (quincy) 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:election_strategy: 1 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:33:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-07-26T23:33:19.538 DEBUG:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-07-26T23:33:19.553 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3b7b1763-526e-44bf-8f07-302d3301e114... 2024-07-26T23:33:19.555 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-07-26T23:33:19.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-07-26T23:33:19.637 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2024-07-26T23:33:19.637 INFO:teuthology.orchestra.run.smithi139.stdout: chdir = "" 2024-07-26T23:33:19.637 INFO:teuthology.orchestra.run.smithi139.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-07-26T23:33:19.637 INFO:teuthology.orchestra.run.smithi139.stdout: auth supported = cephx 2024-07-26T23:33:19.637 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: filestore xattr use omap = true 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: mon clock drift allowed = 1.000 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: osd crush chooseleaf type = 0 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: auth debug = true 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on old message = true 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on bug = true 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: mon max pg per osd = 10000 # >= luminous 2024-07-26T23:33:19.638 INFO:teuthology.orchestra.run.smithi139.stdout: mon pg warn max object skew = 0 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: # disable pg_autoscaler by default for new pools 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: osd_pool_default_pg_autoscale_mode = off 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default size = 2 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow primary affinity = true 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow pg remap = true 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on legacy crush tunables = false 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on crush straw calc version zero = false 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on no sortbitwise = false 2024-07-26T23:33:19.639 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on osd down out interval zero = false 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on too few osds = false 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_no_redundancy = false 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: mon_allow_pool_size_one = true 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: osd default data pool replay window = 5 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: mon allow pool delete = true 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: mon cluster log file level = debug 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: debug asserts on shutdown = true 2024-07-26T23:33:19.640 INFO:teuthology.orchestra.run.smithi139.stdout: mon health detail to clog = false 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: mon host = "172.21.15.139,[v2:172.21.15.139:3301,v1:172.21.15.139:6790],172.21.15.177" 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: mon client directed command retry = 5 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: mon election default strategy = 1 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: ms inject socket failures = 5000 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: ms type = async 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: osd heartbeat grace = 160 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: osd op thread timeout = 150 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout:[osd] 2024-07-26T23:33:19.641 INFO:teuthology.orchestra.run.smithi139.stdout: osd journal size = 100 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub load threshold = 5.0 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub max interval = 600 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: osd mclock profile = high_recovery_ops 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: osd recover clone overlap = true 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: osd recovery max chunk = 1048576 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.642 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug shutdown = true 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug op order = true 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify stray on activate = true 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug trim objects = true 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: osd open classes on start = true 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug pg log writeout = true 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: osd deep scrub update digest min age = 30 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: osd map max advance = 10 2024-07-26T23:33:19.643 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: journal zero on create = true 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: filestore ondisk finisher threads = 3 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: filestore apply finisher threads = 3 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: bdev debug aio = true 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug misdirected ops = true 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: bdev async discard = True 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: bdev enable discard = True 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore allocator = bitmap 2024-07-26T23:33:19.644 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore block size = 96636764160 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore fsck on mount = True 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore zero block detection = True 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: debug bluefs = 1 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: debug bluestore = 1 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: debug monc = 20 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: debug osd = 20 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: debug rocksdb = 10 2024-07-26T23:33:19.645 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd backfillfull_ratio = 0.85 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd full ratio = 0.9 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd nearfull ratio = 0.8 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd class default list = * 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd class load list = * 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd client watch timeout = 120 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify cached snaps = True 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify missing on start = True 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd failsafe full ratio = 0.95 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd fast shutdown = False 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd mclock override recovery settings = True 2024-07-26T23:33:19.646 INFO:teuthology.orchestra.run.smithi139.stdout: osd objectstore = bluestore 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: osd op queue = debug_random 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: osd op queue cut off = debug_random 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: osd shutdown pgref assert = True 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr] 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: debug mgr = 20 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: mgr/telemetry/nag = false 2024-07-26T23:33:19.647 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout:[mon] 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: debug paxos = 20 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: mon data avail warn = 5 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: mon mgr mkfs grace = 240 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd reporter subtree level = osd 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd prime pg temp = true 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.648 INFO:teuthology.orchestra.run.smithi139.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: auth mon ticket ttl = 660 # 11m 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: auth service ticket ttl = 240 # 4m 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: # don't complain about insecure global_id in the test suite 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: # 1m isn't quite enough 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: mon_down_mkfs_grace = 2m 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_filestore_osds = false 2024-07-26T23:33:19.649 INFO:teuthology.orchestra.run.smithi139.stdout: mon min osdmap epochs = 50 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd crush smoke test = False 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: mon osdmap full prune interval = 2 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: mon osdmap full prune min = 15 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: mon osdmap full prune txsize = 2 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: mon scrub interval = 300 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on pool no app = False 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: paxos service trim min = 10 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout:[client] 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: rgw cache enabled = true 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable ops log = true 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable usage log = true 2024-07-26T23:33:19.650 INFO:teuthology.orchestra.run.smithi139.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout: debug monc = 20 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout: debug objecter = 20 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout: debug rados = 20 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.a] 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.c] 2024-07-26T23:33:19.651 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.b] 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.652 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-07-26T23:33:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-07-26T23:33:19.654 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-07-26T23:33:19.655 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.139,[v2:172.21.15.139:3301,v1:172.21.15.139:6790],172.21.15.177" 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 5000 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: ms type = async 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: osd heartbeat grace = 160 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: osd op thread timeout = 150 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-07-26T23:33:19.656 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-07-26T23:33:19.657 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-07-26T23:33:19.658 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2024-07-26T23:33:19.659 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-07-26T23:33:19.660 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-07-26T23:33:19.660 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-07-26T23:33:19.660 INFO:teuthology.orchestra.run.smithi177.stdout: osd class default list = * 2024-07-26T23:33:19.660 INFO:teuthology.orchestra.run.smithi177.stdout: osd class load list = * 2024-07-26T23:33:19.660 INFO:teuthology.orchestra.run.smithi177.stdout: osd client watch timeout = 120 2024-07-26T23:33:19.660 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2024-07-26T23:33:19.660 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2024-07-26T23:33:19.661 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-07-26T23:33:19.661 INFO:teuthology.orchestra.run.smithi177.stdout: osd fast shutdown = False 2024-07-26T23:33:19.661 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2024-07-26T23:33:19.661 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-07-26T23:33:19.661 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: osd shutdown pgref assert = True 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.662 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-07-26T23:33:19.663 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-07-26T23:33:19.664 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-07-26T23:33:19.664 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.664 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-07-26T23:33:19.664 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-07-26T23:33:19.664 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-07-26T23:33:19.664 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.665 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon min osdmap epochs = 50 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd crush smoke test = False 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune interval = 2 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune min = 15 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune txsize = 2 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on pool no app = False 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: paxos service trim min = 10 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-07-26T23:33:19.666 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: debug objecter = 20 2024-07-26T23:33:19.667 INFO:teuthology.orchestra.run.smithi177.stdout: debug rados = 20 2024-07-26T23:33:19.668 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-07-26T23:33:19.668 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-07-26T23:33:19.668 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-07-26T23:33:19.670 INFO:tasks.ceph:Creating admin key on mon.a... 2024-07-26T23:33:19.670 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-07-26T23:33:19.764 INFO:tasks.ceph:Copying monmap to all nodes... 2024-07-26T23:33:19.764 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:19.765 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-07-26T23:33:19.780 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:19.780 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-07-26T23:33:19.835 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi139.front.sepia.ceph.com 2024-07-26T23:33:19.835 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:19.836 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.keyring 2024-07-26T23:33:19.836 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-26T23:33:19.910 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:19.910 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:19.966 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-07-26T23:33:19.966 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:19.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-07-26T23:33:19.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-26T23:33:20.004 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:20.004 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:20.060 INFO:tasks.ceph:Setting up mon nodes... 2024-07-26T23:33:20.060 INFO:tasks.ceph:Setting up mgr nodes... 2024-07-26T23:33:20.061 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-07-26T23:33:20.111 INFO:teuthology.orchestra.run.smithi139.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-07-26T23:33:20.114 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-07-26T23:33:20.177 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-07-26T23:33:20.180 INFO:tasks.ceph:Setting up mds nodes... 2024-07-26T23:33:20.180 INFO:tasks.ceph_client:Setting up client nodes... 2024-07-26T23:33:20.180 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T23:33:20.221 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-07-26T23:33:20.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-07-26T23:33:20.275 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-07-26T23:33:20.287 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-07-26T23:33:20.287 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-07-26T23:33:20.287 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-07-26T23:33:20.312 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:20.312 INFO:tasks.ceph:role: osd.0 2024-07-26T23:33:20.312 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-26T23:33:20.312 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:20.385 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:20.898 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-07-26T23:33:20.900 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-07-26T23:33:20.900 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-07-26T23:33:21.015 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-07-26T23:33:21.048 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-07-26T23:33:21.113 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:21.113 INFO:tasks.ceph:role: osd.1 2024-07-26T23:33:21.114 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-26T23:33:21.114 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-07-26T23:33:21.185 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:21.186 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:21.649 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-07-26T23:33:21.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-07-26T23:33:21.651 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-07-26T23:33:21.741 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-07-26T23:33:21.771 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-07-26T23:33:21.836 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:21.836 INFO:tasks.ceph:role: osd.2 2024-07-26T23:33:21.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-26T23:33:21.837 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-07-26T23:33:21.909 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:21.909 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:21.909 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:21.909 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:21.909 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:21.909 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:21.910 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:21.910 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:21.910 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:21.910 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:22.396 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-07-26T23:33:22.398 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-07-26T23:33:22.398 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-07-26T23:33:22.527 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-07-26T23:33:22.557 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-07-26T23:33:22.623 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:22.623 INFO:tasks.ceph:role: osd.3 2024-07-26T23:33:22.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-26T23:33:22.623 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-07-26T23:33:22.696 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:22.696 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:22.696 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:22.696 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:22.696 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:22.696 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:22.696 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:22.697 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:22.697 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:22.697 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:23.205 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2024-07-26T23:33:23.207 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2024-07-26T23:33:23.207 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-07-26T23:33:23.339 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-07-26T23:33:23.373 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:23.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:23.467+0000 7fe062a2e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-07-26T23:33:23.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:23.468+0000 7fe062a2e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-07-26T23:33:23.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:23.468+0000 7fe062a2e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-26T23:33:23.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:23.468+0000 7fe062a2e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-26T23:33:23.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:23.468+0000 7fe062a2e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-26T23:33:23.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:23.468+0000 7fe062a2e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-07-26T23:33:26.308 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-07-26T23:33:26.337 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:26.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:26.426+0000 7fd714f99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-07-26T23:33:26.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:26.426+0000 7fd714f99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-07-26T23:33:26.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:26.426+0000 7fd714f99640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-26T23:33:26.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:26.426+0000 7fd714f99640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-26T23:33:26.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:26.426+0000 7fd714f99640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-26T23:33:26.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:26.427+0000 7fd714f99640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-07-26T23:33:29.266 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-07-26T23:33:29.297 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:29.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:29.386+0000 7fc580218640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-07-26T23:33:29.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:29.386+0000 7fc580218640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-07-26T23:33:29.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:29.386+0000 7fc580218640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-26T23:33:29.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:29.386+0000 7fc580218640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-26T23:33:29.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:29.386+0000 7fc580218640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-26T23:33:29.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:29.387+0000 7fc580218640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-07-26T23:33:32.244 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-07-26T23:33:32.271 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:32.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:32.359+0000 7f3e42410640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-07-26T23:33:32.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:32.360+0000 7f3e42410640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-07-26T23:33:32.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:32.360+0000 7f3e42410640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-26T23:33:32.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:32.360+0000 7f3e42410640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-26T23:33:32.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:32.360+0000 7f3e42410640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-26T23:33:32.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:32.360+0000 7f3e42410640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-07-26T23:33:35.207 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-07-26T23:33:35.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-07-26T23:33:35.266 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:35.266 INFO:tasks.ceph:role: osd.4 2024-07-26T23:33:35.266 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-26T23:33:35.266 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-07-26T23:33:35.342 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:35.342 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:35.342 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:35.343 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:35.343 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:35.343 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:35.343 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:35.343 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:35.343 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:35.343 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:35.857 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-07-26T23:33:35.860 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-07-26T23:33:35.860 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-07-26T23:33:35.970 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-07-26T23:33:36.005 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-07-26T23:33:36.070 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:36.070 INFO:tasks.ceph:role: osd.5 2024-07-26T23:33:36.070 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-26T23:33:36.071 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-07-26T23:33:36.143 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:36.144 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:36.144 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:36.144 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:36.144 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:36.144 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:36.145 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:36.145 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:36.145 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:36.145 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:36.586 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-07-26T23:33:36.588 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-07-26T23:33:36.589 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-07-26T23:33:36.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-07-26T23:33:36.741 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-07-26T23:33:36.805 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:36.806 INFO:tasks.ceph:role: osd.6 2024-07-26T23:33:36.806 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-26T23:33:36.806 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-07-26T23:33:36.878 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:36.879 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:37.361 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-07-26T23:33:37.363 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-07-26T23:33:37.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-07-26T23:33:37.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-07-26T23:33:37.527 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-07-26T23:33:37.593 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T23:33:37.593 INFO:tasks.ceph:role: osd.7 2024-07-26T23:33:37.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-26T23:33:37.593 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-07-26T23:33:37.666 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T23:33:37.666 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T23:33:37.666 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T23:33:37.666 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T23:33:37.666 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T23:33:37.666 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-07-26T23:33:37.667 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T23:33:37.667 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T23:33:37.667 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T23:33:37.667 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T23:33:38.202 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-07-26T23:33:38.220 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-07-26T23:33:38.220 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-07-26T23:33:38.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-07-26T23:33:38.370 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:38.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:38.474+0000 7f7503001640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-07-26T23:33:38.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:38.474+0000 7f7503001640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-07-26T23:33:38.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:38.474+0000 7f7503001640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-26T23:33:38.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:38.474+0000 7f7503001640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-26T23:33:38.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:38.474+0000 7f7503001640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-26T23:33:38.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:38.475+0000 7f7503001640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-07-26T23:33:41.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-07-26T23:33:41.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:41.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:41.450+0000 7fa65bc13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-07-26T23:33:41.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:41.450+0000 7fa65bc13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-07-26T23:33:41.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:41.450+0000 7fa65bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-26T23:33:41.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:41.450+0000 7fa65bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-26T23:33:41.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:41.450+0000 7fa65bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-26T23:33:41.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:41.451+0000 7fa65bc13640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-07-26T23:33:44.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-07-26T23:33:44.326 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:44.417 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:44.415+0000 7f355b815640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-07-26T23:33:44.417 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:44.415+0000 7f355b815640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-07-26T23:33:44.417 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:44.415+0000 7f355b815640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-26T23:33:44.417 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:44.415+0000 7f355b815640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-26T23:33:44.417 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:44.415+0000 7f355b815640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-26T23:33:44.417 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:44.416+0000 7f355b815640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-07-26T23:33:47.261 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-07-26T23:33:47.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:47.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:47.369+0000 7f94c03b8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-07-26T23:33:47.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:47.370+0000 7f94c03b8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-07-26T23:33:47.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:47.370+0000 7f94c03b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-26T23:33:47.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:47.370+0000 7f94c03b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-26T23:33:47.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:47.370+0000 7f94c03b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-26T23:33:47.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:33:47.370+0000 7f94c03b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-07-26T23:33:50.208 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-07-26T23:33:50.237 INFO:tasks.ceph:Reading keys from all nodes... 2024-07-26T23:33:50.237 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:50.237 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-07-26T23:33:50.267 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:50.268 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-07-26T23:33:50.338 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:50.339 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-07-26T23:33:50.406 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:50.406 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-07-26T23:33:50.474 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:50.474 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-07-26T23:33:50.543 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:50.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-07-26T23:33:50.574 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:50.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-07-26T23:33:50.642 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:50.643 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-07-26T23:33:50.711 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:50.711 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-07-26T23:33:50.778 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:50.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-07-26T23:33:50.845 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:33:50.845 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-07-26T23:33:50.863 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:33:50.863 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-07-26T23:33:50.902 INFO:tasks.ceph:Adding keys to all mons... 2024-07-26T23:33:50.902 DEBUG:teuthology.orchestra.run.smithi139:> sudo tee -a /etc/ceph/ceph.keyring 2024-07-26T23:33:50.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-07-26T23:33:50.946 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.y] 2024-07-26T23:33:50.946 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBAMqRmR3ieBhAAZJDayXMvKVyFs+8TvFY0/g== 2024-07-26T23:33:50.949 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBDMqRmN8/0GxAA0p1GYTV2nYE2LDOHNHqiRg== 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBGMqRm2mJ7GRAAxqOZfHiPplJGVrJWQVkNjw== 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBJMqRmCGQaFxAAcG6GrBRKKV/kyFNHCMndeA== 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.3] 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBMMqRmCSWFFRAA9W2Luw/i1K/CmO8tTA73tQ== 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.x] 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBAMqRmSi+QChAA9FkRCDY9I0cpOItcQe66gg== 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.4] 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBSMqRmQTVTHBAAc2hcqG0znMlL7Rmg6+bd0w== 2024-07-26T23:33:50.950 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.5] 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBVMqRmnVbnGhAAX/Iw9yrNoHmLtBKUFMJ3Nw== 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.6] 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBYMqRm5vLSGBAAhzkkQqbWb5Jy+t1Q0YXoEw== 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.7] 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBbMqRmYV4ZFhAACjmvYIo5TDjtCth4mix8gw== 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBAMqRmsFcsDRAADEMAgwpcaDbh1nMUXURmjw== 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout:[client.1] 2024-07-26T23:33:50.951 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBAMqRmhtFgEBAAMYnvIHcIPu7rwIc5JoBVsw== 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.y] 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBAMqRmR3ieBhAAZJDayXMvKVyFs+8TvFY0/g== 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBDMqRmN8/0GxAA0p1GYTV2nYE2LDOHNHqiRg== 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBGMqRm2mJ7GRAAxqOZfHiPplJGVrJWQVkNjw== 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBJMqRmCGQaFxAAcG6GrBRKKV/kyFNHCMndeA== 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBMMqRmCSWFFRAA9W2Luw/i1K/CmO8tTA73tQ== 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-07-26T23:33:50.967 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBAMqRmSi+QChAA9FkRCDY9I0cpOItcQe66gg== 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBSMqRmQTVTHBAAc2hcqG0znMlL7Rmg6+bd0w== 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBVMqRmnVbnGhAAX/Iw9yrNoHmLtBKUFMJ3Nw== 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.6] 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYMqRm5vLSGBAAhzkkQqbWb5Jy+t1Q0YXoEw== 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.7] 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBbMqRmYV4ZFhAACjmvYIo5TDjtCth4mix8gw== 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBAMqRmsFcsDRAADEMAgwpcaDbh1nMUXURmjw== 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2024-07-26T23:33:50.968 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBAMqRmhtFgEBAAMYnvIHcIPu7rwIc5JoBVsw== 2024-07-26T23:33:50.969 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T23:33:50.995 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T23:33:51.064 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.117 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.170 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.173 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.223 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.226 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.277 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T23:33:51.279 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T23:33:51.332 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.385 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.438 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.491 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.494 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T23:33:51.545 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T23:33:51.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T23:33:51.601 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T23:33:51.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T23:33:51.656 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-07-26T23:33:51.657 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-07-26T23:33:51.682 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-26T23:33:51.779 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-07-26T23:33:51.804 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-07-26T23:33:51.870 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-26T23:33:51.966 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-07-26T23:33:51.990 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-07-26T23:33:52.015 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-26T23:33:52.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-07-26T23:33:52.137 DEBUG:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:52.140 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-07-26T23:33:52.192 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-07-26T23:33:52.193 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-07-26T23:33:52.193 INFO:tasks.ceph.mon.a:Restarting daemon 2024-07-26T23:33:52.193 DEBUG:teuthology.orchestra.run.smithi139:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-07-26T23:33:52.196 INFO:tasks.ceph.mon.a:Started 2024-07-26T23:33:52.196 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-07-26T23:33:52.196 INFO:tasks.ceph.mon.c:Restarting daemon 2024-07-26T23:33:52.196 DEBUG:teuthology.orchestra.run.smithi139:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-07-26T23:33:52.198 INFO:tasks.ceph.mon.c:Started 2024-07-26T23:33:52.198 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-07-26T23:33:52.198 INFO:tasks.ceph.mon.b:Restarting daemon 2024-07-26T23:33:52.198 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-07-26T23:33:52.235 INFO:tasks.ceph.mon.b:Started 2024-07-26T23:33:52.236 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-07-26T23:33:52.236 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-07-26T23:33:52.236 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-07-26T23:33:52.238 INFO:tasks.ceph.mgr.y:Started 2024-07-26T23:33:52.238 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-07-26T23:33:52.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-07-26T23:33:52.240 INFO:tasks.ceph.mgr.x:Started 2024-07-26T23:33:52.241 DEBUG:tasks.ceph:set 0 configs 2024-07-26T23:33:52.241 INFO:tasks.ceph:Setting crush tunables to default 2024-07-26T23:33:52.241 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster ceph osd crush tunables default 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.339+0000 7fb074668640 10 monclient: get_monmap_and_config 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.339+0000 7fb074668640 10 monclient: build_initial_monmap 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.339+0000 7fb074668640 10 monclient: monmap: 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:52.340443+0000 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:52.340443+0000 2024-07-26T23:33:52.341 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.339+0000 7fb074668640 1 Processor -- start 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 1 -- start start 2024-07-26T23:33:52.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient: init 2024-07-26T23:33:52.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient: _reopen_session rank -1 2024-07-26T23:33:52.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-07-26T23:33:52.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb06c1470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:33:52.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 0x7fb06c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:33:52.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient(hunting): picked mon.noname-a con 0x7fb06c0b41c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb06c14c1d0 addr v1:172.21.15.177:6789/0 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb06c147ce0 con 0x7fb06c1470c0 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb06c0acaf0 con 0x7fb06c14c1d0 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient(hunting): start opening mon connection 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb06c0b41c0 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient(hunting): _renew_subs 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb072404640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb06c1470c0 legacy=0x7fb06c140950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb074668640 10 monclient(hunting): authenticate will time out at 1265.376892s 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb071c03640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 msgr2=0x7fb06c0b45e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb071c03640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 0x7fb06c0b45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:33:52.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.340+0000 7fb072c05640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7fb06c14c1d0 legacy=0x7fb06c14e5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-07-26T23:33:52.542 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.541+0000 7fb072404640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb06c1470c0 legacy=0x7fb06c140950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:52.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.541+0000 7fb071c03640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 msgr2=0x7fb06c0b45e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:52.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.541+0000 7fb071c03640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 0x7fb06c0b45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:33:52.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.541+0000 7fb072c05640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7fb06c14c1d0 legacy=0x7fb06c14e5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-07-26T23:33:52.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.942+0000 7fb072404640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb06c1470c0 legacy=0x7fb06c140950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:52.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.942+0000 7fb071c03640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 msgr2=0x7fb06c0b45e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:52.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.942+0000 7fb071c03640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 0x7fb06c0b45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:33:52.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:52.942+0000 7fb072c05640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7fb06c14c1d0 legacy=0x7fb06c14e5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-07-26T23:33:53.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:53.743+0000 7fb072404640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb06c1470c0 legacy=0x7fb06c140950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:53.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:53.743+0000 7fb071c03640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 msgr2=0x7fb06c0b45e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:53.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:53.743+0000 7fb071c03640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 0x7fb06c0b45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:33:53.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:53.743+0000 7fb072c05640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7fb06c14c1d0 legacy=0x7fb06c14e5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.177:6789/0 2024-07-26T23:33:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): tick 2024-07-26T23:33:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): _check_auth_tickets 2024-07-26T23:33:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 monclient(hunting): continuing hunt 2024-07-26T23:33:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): _reopen_session rank -1 2024-07-26T23:33:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 msgr2=0x7fb06c0b45e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:33:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb06c0b41c0 0x7fb06c0b45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 -- >> v1:172.21.15.177:6789/0 conn(0x7fb06c14c1d0 legacy=0x7fb06c14e5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb06c1470c0 legacy=0x7fb06c140950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient: _add_conns ranks=[2,4] 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 0x7fb050005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb0500036a0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 0x7fb0500085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb0500060e0 addr v2:172.21.15.177:3300/0 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb0500060e0 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb050008c60 con 0x7fb0500036a0 2024-07-26T23:33:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.340+0000 7fb063fff640 10 monclient(hunting): _renew_subs 2024-07-26T23:33:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.341+0000 7fb072404640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 msgr2=0x7fb050005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.341+0000 7fb072404640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 0x7fb050005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:33:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.341+0000 7fb071c03640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 msgr2=0x7fb0500085a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-07-26T23:33:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.341+0000 7fb071c03640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 0x7fb0500085a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:33:55.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.542+0000 7fb072404640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 msgr2=0x7fb050005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:55.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.542+0000 7fb072404640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 0x7fb050005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:33:55.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.542+0000 7fb071c03640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 msgr2=0x7fb0500085a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-07-26T23:33:55.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.542+0000 7fb071c03640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 0x7fb0500085a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:33:55.945 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.944+0000 7fb072404640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 msgr2=0x7fb050005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:55.945 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.944+0000 7fb072404640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 0x7fb050005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:33:55.945 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.944+0000 7fb071c03640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 msgr2=0x7fb0500085a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-07-26T23:33:55.946 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:55.944+0000 7fb071c03640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 0x7fb0500085a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:33:56.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:56.746+0000 7fb072404640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 msgr2=0x7fb050005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:56.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:56.746+0000 7fb072404640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 0x7fb050005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:33:56.748 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:56.746+0000 7fb071c03640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 msgr2=0x7fb0500085a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.177:3300/0 2024-07-26T23:33:56.748 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:56.746+0000 7fb071c03640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 0x7fb0500085a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:33:58.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): tick 2024-07-26T23:33:58.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): _check_auth_tickets 2024-07-26T23:33:58.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 monclient(hunting): continuing hunt 2024-07-26T23:33:58.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): _reopen_session rank -1 2024-07-26T23:33:58.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 msgr2=0x7fb050005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:33:58.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500036a0 0x7fb050005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:33:58.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 -- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 msgr2=0x7fb0500085a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:33:58.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0500060e0 0x7fb0500085a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:33:58.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb072c05640 1 -- reap_dead start 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 0x7fb050006590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb050003ae0 addr v2:172.21.15.139:3300/0 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb050006ad0 addr v1:172.21.15.139:6789/0 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 0x7fb05000d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb0500085f0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb06c1474a0 con 0x7fb050006ad0 2024-07-26T23:33:58.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:33:58.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb050003ae0 2024-07-26T23:33:58.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:33:58.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb050008c60 con 0x7fb0500085f0 2024-07-26T23:33:58.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb063fff640 10 monclient(hunting): _renew_subs 2024-07-26T23:33:58.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb072404640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 msgr2=0x7fb050006590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:58.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb071c03640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb050006ad0 legacy=0x7fb050006ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:58.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb072c05640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 msgr2=0x7fb05000d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:58.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb072404640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 0x7fb050006590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:33:58.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.341+0000 7fb072c05640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 0x7fb05000d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:33:58.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.542+0000 7fb072404640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 msgr2=0x7fb050006590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:58.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.542+0000 7fb071c03640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb050006ad0 legacy=0x7fb050006ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:58.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.542+0000 7fb072c05640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 msgr2=0x7fb05000d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:58.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.542+0000 7fb072404640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 0x7fb050006590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:33:58.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.542+0000 7fb072c05640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 0x7fb05000d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:33:58.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.943+0000 7fb071c03640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb050006ad0 legacy=0x7fb050006ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:58.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.943+0000 7fb072404640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 msgr2=0x7fb050006590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:58.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.943+0000 7fb072c05640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 msgr2=0x7fb05000d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:58.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.943+0000 7fb072c05640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 0x7fb05000d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:33:58.945 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:58.943+0000 7fb072404640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 0x7fb050006590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:33:59.300 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:00:06.963 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:33:59.606 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:00:07.268 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:33:59.650 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:00:07.321 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:33:59.652 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:00:07.324 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:33:59.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:59.745+0000 7fb071c03640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb050006ad0 legacy=0x7fb050006ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.139:6789/0 2024-07-26T23:33:59.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:59.745+0000 7fb072404640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 msgr2=0x7fb050006590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3300/0 2024-07-26T23:33:59.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:59.745+0000 7fb072c05640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 msgr2=0x7fb05000d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.139:3301/0 2024-07-26T23:33:59.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:59.745+0000 7fb072404640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 0x7fb050006590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:33:59.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:33:59.745+0000 7fb072c05640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 0x7fb05000d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:33:59.972 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:00:07.643 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:33:59.974 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:00:07.645 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient(hunting): tick 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient(hunting): _check_auth_tickets 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 monclient(hunting): continuing hunt 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient(hunting): _reopen_session rank -1 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 -- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 msgr2=0x7fb05000d030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb0500085f0 0x7fb05000d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 -- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 msgr2=0x7fb050006590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb050003ae0 0x7fb050006590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 -- >> v1:172.21.15.139:6789/0 conn(0x7fb050006ad0 legacy=0x7fb050006ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:01.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient: _add_conns ranks=[1,4] 2024-07-26T23:34:01.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb05000d7b0 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:01.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:01.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb050011970 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:01.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb06c147ce0 con 0x7fb05000d7b0 2024-07-26T23:34:01.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:01.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb050011970 2024-07-26T23:34:01.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.341+0000 7fb063fff640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:01.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.379+0000 7fb071c03640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:01.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.382+0000 7fb071c03640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:40056/0 (socket says 172.21.15.139:40056) 2024-07-26T23:34:01.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.382+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 learned_addr learned my addr 172.21.15.139:0/2553863220 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:01.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.382+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:01.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.382+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:01.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.382+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:01.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.382+0000 7fb071c03640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:01.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.384+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T23:34:01.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.384+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:01.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.384+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:01.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.384+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:34:01.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.586+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:01.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.586+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:01.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.586+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:01.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.586+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:01.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.586+0000 7fb071c03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:01.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.587+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T23:34:01.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.587+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:01.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.587+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:01.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.587+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:34:01.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.989+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:01.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.989+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:01.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.989+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:01.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.989+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:01.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.989+0000 7fb071c03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:01.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.990+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T23:34:01.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.990+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:01.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.990+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:01.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:01.990+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:34:02.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.792+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:02.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.792+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:02.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.792+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:02.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.792+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:02.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.792+0000 7fb071c03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:02.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.793+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T23:34:02.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.793+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:02.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.793+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:02.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:02.793+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:34:04.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): tick 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): _check_auth_tickets 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 monclient(hunting): continuing hunt 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): _reopen_session rank -1 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v1:172.21.15.177:6789/0 conn(0x7fb05000d7b0 legacy=0x7fb05000dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-07-26T23:34:04.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 reap_dead start 2024-07-26T23:34:04.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:04.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb050011970 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:04.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb050006ad0 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:04.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb050006590 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:04.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb06c1474a0 con 0x7fb050006ad0 2024-07-26T23:34:04.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb06c0acaf0 con 0x7fb050006590 2024-07-26T23:34:04.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:04.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb050011970 2024-07-26T23:34:04.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.341+0000 7fb063fff640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:04.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.354+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:04.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.355+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:04.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.355+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:04.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.355+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:04.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.355+0000 7fb072404640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:04.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.357+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:04.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.357+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:04.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.357+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:04.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.357+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:34:04.560 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.559+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:04.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.559+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:04.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.559+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:04.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.559+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:04.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.559+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:04.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.560+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:04.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.560+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:04.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.560+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:04.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.560+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:34:04.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.961+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:04.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:04.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:04.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:04.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:04.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:04.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:04.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:04.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:04.962+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:34:05.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.764+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:05.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.764+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:05.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.764+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:05.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.764+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:05.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.764+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:05.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.765+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:05.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.765+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:05.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.765+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:05.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:05.765+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:34:07.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): tick 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): _check_auth_tickets 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 monclient(hunting): continuing hunt 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): _reopen_session rank -1 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v1:172.21.15.177:6789/0 conn(0x7fb050006590 legacy=0x7fb05000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v1:172.21.15.139:6789/0 conn(0x7fb050006ad0 legacy=0x7fb050006ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient: _add_conns ranks=[2,3] 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb05000c030 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:07.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb050010ea0 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:07.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:07.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb050010ea0 2024-07-26T23:34:07.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:07.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb050008c60 con 0x7fb05000c030 2024-07-26T23:34:07.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb063fff640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:07.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:07.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb05000c030 auth_method 0 2024-07-26T23:34:07.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:07.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:07.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.342+0000 7fb072404640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:07.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.343+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:07.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.343+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:07.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.343+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:07.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.343+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:34:07.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.357+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:07.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.357+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050010ea0 auth_method 0 2024-07-26T23:34:07.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.357+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:07.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.357+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:07.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.357+0000 7fb071c03640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:07.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.358+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T23:34:07.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.358+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:07.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.358+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:07.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.358+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:34:07.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.544+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:07.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.545+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb05000c030 auth_method 0 2024-07-26T23:34:07.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.545+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:07.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.545+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:07.546 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.545+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:07.547 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.546+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:07.547 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.546+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:07.547 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.546+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:07.547 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.546+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:34:07.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.560+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:07.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.560+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050010ea0 auth_method 0 2024-07-26T23:34:07.561 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.560+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:07.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.560+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:07.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.560+0000 7fb071c03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:07.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.561+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:07.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.561+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:07.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.561+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:07.562 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.561+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:34:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.948+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.948+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb05000c030 auth_method 0 2024-07-26T23:34:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.948+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.948+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.948+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:07.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.950+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:07.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.950+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:07.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.950+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:07.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:07.950+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:34:08.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.648+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:08.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.649+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050010ea0 auth_method 0 2024-07-26T23:34:08.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.649+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:08.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.649+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:08.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.649+0000 7fb071c03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:08.654 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.653+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:08.654 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.653+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:08.654 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.653+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:08.654 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.653+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:34:08.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.752+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:08.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.754+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb05000c030 auth_method 0 2024-07-26T23:34:08.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.754+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:08.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.754+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:08.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.754+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:08.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.755+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:08.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.755+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:08.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.755+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:08.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:08.755+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:34:09.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.456+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:09.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.456+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb050010ea0 auth_method 0 2024-07-26T23:34:09.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.456+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:09.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.456+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:09.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.456+0000 7fb071c03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:09.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.457+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:09.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.457+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:09.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.457+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:09.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:09.457+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:34:10.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): tick 2024-07-26T23:34:10.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): _check_auth_tickets 2024-07-26T23:34:10.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 monclient(hunting): continuing hunt 2024-07-26T23:34:10.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): _reopen_session rank -1 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 msgr2=0x7fb050010960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb05000c030 0x7fb050010960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010ea0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 reap_dead start 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb050011970 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb050006590 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb050006ad0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:10.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb050005ac0 con 0x7fb050006590 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb050011970 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb050008c60 con 0x7fb050006ad0 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb063fff640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.342+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072c05640 10 monclient(hunting): get_auth_request con 0x7fb050006ad0 auth_method 0 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072c05640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072c05640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:10.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:10.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:10.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:10.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:10.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.343+0000 7fb072404640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:10.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.344+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:34:10.548 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072c05640 10 monclient(hunting): get_auth_request con 0x7fb050006ad0 auth_method 0 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072c05640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:10.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.546+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:10.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.547+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:34:10.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.948+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:10.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.948+0000 7fb072c05640 10 monclient(hunting): get_auth_request con 0x7fb050006ad0 auth_method 0 2024-07-26T23:34:10.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.948+0000 7fb072c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:10.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.948+0000 7fb072c05640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:10.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.948+0000 7fb072c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:10.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:10.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:10.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:10.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:10.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:10.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:10.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:10.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:10.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.949+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:34:10.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.950+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T23:34:10.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.950+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:10.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.950+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:10.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:10.950+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:34:11.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.751+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:11.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.751+0000 7fb072c05640 10 monclient(hunting): get_auth_request con 0x7fb050006ad0 auth_method 0 2024-07-26T23:34:11.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.751+0000 7fb072c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:11.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.751+0000 7fb072c05640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:11.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.751+0000 7fb072c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:11.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.751+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:11.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:11.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:11.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:11.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050011970 auth_method 0 2024-07-26T23:34:11.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:11.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:11.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:11.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.752+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:34:11.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.753+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-26T23:34:11.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.753+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:11.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.753+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:11.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:11.753+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:34:13.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient(hunting): tick 2024-07-26T23:34:13.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient(hunting): _check_auth_tickets 2024-07-26T23:34:13.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 monclient(hunting): continuing hunt 2024-07-26T23:34:13.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient(hunting): _reopen_session rank -1 2024-07-26T23:34:13.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 msgr2=0x7fb050011300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:13.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb050006ad0 0x7fb050011300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 msgr2=0x7fb050013d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.177:3300/0 conn(0x7fb050011970 0x7fb050013d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 >> v1:172.21.15.139:6789/0 conn(0x7fb050006590 legacy=0x7fb05000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient: _add_conns ranks=[3,1] 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb050010960 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb050010e00 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb050005eb0 con 0x7fb050010e00 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 1 -- 172.21.15.139:0/2553863220 --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb050010960 2024-07-26T23:34:13.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.342+0000 7fb063fff640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:13.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.344+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:13.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.344+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050010960 auth_method 0 2024-07-26T23:34:13.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.344+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:13.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.344+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:13.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.344+0000 7fb072404640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:13.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.345+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:13.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.345+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:13.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.345+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:13.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.345+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T23:34:13.548 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.546+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:13.548 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.547+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050010960 auth_method 0 2024-07-26T23:34:13.548 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.547+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:13.548 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.547+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:13.548 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.547+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:13.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.548+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:13.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.548+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:13.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.548+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:13.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.548+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T23:34:13.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.975+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:13.977 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.975+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050010960 auth_method 0 2024-07-26T23:34:13.977 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.975+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:13.977 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.975+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:13.977 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.975+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:13.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.976+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:13.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.976+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:13.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.976+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:13.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:13.976+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T23:34:14.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.798+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:14.801 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.800+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb050010960 auth_method 0 2024-07-26T23:34:14.801 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.800+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:14.801 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.800+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:14.801 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.800+0000 7fb072404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:14.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.802+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T23:34:14.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.802+0000 7fb072404640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T23:34:14.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.802+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T23:34:14.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:14.803+0000 7fb072404640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T23:34:15.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.023+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796016886 0 0) 0x7fb050005eb0 con 0x7fb050010e00 2024-07-26T23:34:15.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.023+0000 7fb071402640 10 monclient(hunting): _negotiate 2024-07-26T23:34:15.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.023+0000 7fb071402640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:15.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.023+0000 7fb071402640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:15.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.023+0000 7fb071402640 10 monclient(hunting): my global_id is 4102 2024-07-26T23:34:15.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.023+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb058003520 con 0x7fb050010e00 2024-07-26T23:34:15.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 747539413 0 0) 0x7fb058003520 con 0x7fb050010e00 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 msgr2=0x7fb05000b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 --2- 172.21.15.139:0/2553863220 >> v2:172.21.15.139:3300/0 conn(0x7fb050010960 0x7fb05000b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: _finish_hunting 0 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 monclient: found mon.noname-c-legacy 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05000bd90 con 0x7fb050010e00 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: _finish_auth 0 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: _check_auth_tickets 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:15.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3006719369 0 0) 0x7fb05c007660 con 0x7fb050010e00 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: dump: 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:15.180 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 >> v1:172.21.15.177:6789/0 conn(0x7fb050010e00 legacy=0x7fb05000b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb072c05640 1 -- 172.21.15.139:0/2553863220 reap_dead start 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0580039b0 0x7fb058003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): picked mon.b con 0x7fb0580039b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058004800 0x7fb058004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:15.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): picked mon.c con 0x7fb058004800 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:15.182 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb058005290 0x7fb05800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:15.182 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): picked mon.a con 0x7fb058005290 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:15.182 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c1477c0 con 0x7fb058005290 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb05800c1e0 con 0x7fb0580039b0 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb05800c3e0 con 0x7fb058004800 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.175+0000 7fb071402640 10 monclient(hunting): _finish_auth 0 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058004800 0x7fb058004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb058005290 0x7fb05800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb071c03640 10 monclient(hunting): get_auth_request con 0x7fb058004800 auth_method 0 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb071c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb071c03640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:15.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb071c03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb072c05640 10 monclient(hunting): get_auth_request con 0x7fb058005290 auth_method 0 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb072c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb072c05640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.176+0000 7fb072c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.177+0000 7fb072c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.177+0000 7fb072c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.177+0000 7fb072c05640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.177+0000 7fb071c03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.177+0000 7fb071c03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:15.184 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.177+0000 7fb071c03640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-26T23:34:15.185 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2024-07-26T23:34:15.185 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0580039b0 msgr2=0x7fb058003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0580039b0 0x7fb058003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb058005290 msgr2=0x7fb05800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb058005290 0x7fb05800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 10 monclient: _finish_hunting 0 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 1 monclient: found mon.c 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 1 -- 172.21.15.139:0/2553863220 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb058003520 con 0x7fb058004800 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 10 monclient: _finish_auth 0 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 10 monclient: _check_auth_tickets 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb071c03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb074668640 5 monclient: authenticate success, global_id 4102 2024-07-26T23:34:15.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb074668640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:15.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.179+0000 7fb072c05640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb058005290 0x7fb05800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T23:34:15.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.181+0000 7fb071c03640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058004800 0x7fb058004c80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009560 tx=0x7fb058005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ec1e6c81d2a4e6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c034070 con 0x7fb058004800 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 10 monclient: dump: 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:15.222 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb05c02ea10 con 0x7fb058004800 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb074668640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 1 -- 172.21.15.139:0/2553863220 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c007d20 con 0x7fb058004800 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb074668640 4 set_mon_vals no callback set 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.220+0000 7fb071402640 10 monclient: dump: 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:15.223 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.224 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.224 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:15.224 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:15.224 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:15.224 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:15.224 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:15.224 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 10 monclient: shutdown 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 -- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058004800 msgr2=0x7fb058004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058004800 0x7fb058004c80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009560 tx=0x7fb058005830 comp rx=0 tx=0).stop 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 -- 172.21.15.139:0/2553863220 shutdown_connections 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058004800 0x7fb058004c80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb058005290 0x7fb05800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 --2- 172.21.15.139:0/2553863220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0580039b0 0x7fb058003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 -- 172.21.15.139:0/2553863220 >> 172.21.15.139:0/2553863220 conn(0x7fb06c0b62c0 msgr2=0x7fb06c0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 -- 172.21.15.139:0/2553863220 shutdown_connections 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.221+0000 7fb074668640 1 -- 172.21.15.139:0/2553863220 wait complete. 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 10 monclient: build_initial_monmap 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 10 monclient: monmap: 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:15.229 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 1 librados: starting msgr at 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 1 librados: starting objecter 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 1 Processor -- start 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 1 -- start start 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 1 librados: setting wanted keys 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 1 librados: calling monclient init 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.222+0000 7fb074668640 10 monclient: init 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0580039b0 0x7fb06c141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): picked mon.noname-a con 0x7fb0580039b0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:15.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb058004800 0x7fb06c1d8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): picked mon.noname-b con 0x7fb058004800 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058005290 0x7fb06c1d93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): picked mon.noname-c con 0x7fb058005290 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c141410 con 0x7fb0580039b0 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c074850 con 0x7fb058004800 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb06c0b45e0 con 0x7fb058005290 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.223+0000 7fb074668640 10 monclient(hunting): authenticate will time out at 1288.259304s 2024-07-26T23:34:15.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.227+0000 7fb072404640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0580039b0 0x7fb06c141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.227+0000 7fb072404640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0580039b0 0x7fb06c141940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36050/0 (socket says 172.21.15.139:36050) 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.227+0000 7fb072404640 1 -- 172.21.15.139:0/1838386011 learned_addr learned my addr 172.21.15.139:0/1838386011 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.227+0000 7fb072404640 10 monclient(hunting): get_auth_request con 0x7fb0580039b0 auth_method 0 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.227+0000 7fb072404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.227+0000 7fb072404640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.227+0000 7fb072404640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.228+0000 7fb072404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:15.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.228+0000 7fb072404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.228+0000 7fb072404640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.228+0000 7fb072c05640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058005290 0x7fb06c1d93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.230+0000 7fb072c05640 10 monclient(hunting): get_auth_request con 0x7fb058005290 auth_method 0 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.230+0000 7fb072c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.230+0000 7fb072c05640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.230+0000 7fb072c05640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 10 monclient(hunting): handle_auth_done global_id 4104 payload 824 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 1 -- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058005290 msgr2=0x7fb06c1d93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058005290 0x7fb06c1d93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 1 -- 172.21.15.139:0/1838386011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb058004800 msgr2=0x7fb06c1d8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb058004800 0x7fb06c1d8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 10 monclient: _finish_hunting 0 2024-07-26T23:34:15.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 1 monclient: found mon.noname-a 2024-07-26T23:34:15.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:15.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:34:15.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.231+0000 7fb072404640 1 -- 172.21.15.139:0/1838386011 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06c143180 con 0x7fb0580039b0 2024-07-26T23:34:15.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.233+0000 7fb072404640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0580039b0 0x7fb06c141940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb068018d90 tx=0x7fb06800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:15.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.233+0000 7fb062ffd640 10 client.?.objecter ms_handle_connect 0x7fb0580039b0 2024-07-26T23:34:15.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.233+0000 7fb062ffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:15.240 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 1 -- 172.21.15.139:0/1838386011 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06800b630 con 0x7fb0580039b0 2024-07-26T23:34:15.240 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:15.240 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:34:15.240 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: dump: 2024-07-26T23:34:15.240 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: _finish_auth 0 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:15.241 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 1 -- 172.21.15.139:0/1838386011 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb06800b7d0 con 0x7fb0580039b0 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 5 monclient: authenticate success, global_id 4104 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 10 monclient: _renew_subs 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06c1433f0 con 0x7fb0580039b0 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 1 -- 172.21.15.139:0/1838386011 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06800bbc0 con 0x7fb0580039b0 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb062ffd640 10 monclient: dump: 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:15.242 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb070c01640 4 set_mon_vals no callback set 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 10 client.4104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 10 monclient: _renew_subs 2024-07-26T23:34:15.243 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:15.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06c14ae00 con 0x7fb0580039b0 2024-07-26T23:34:15.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.238+0000 7fb074668640 1 librados: init done 2024-07-26T23:34:15.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.239+0000 7fb074668640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:15.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.239+0000 7fb074668640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:15.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.239+0000 7fb074668640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:15.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.239+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb06c0b4970 con 0x7fb0580039b0 2024-07-26T23:34:15.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.246+0000 7fb062ffd640 1 -- 172.21.15.139:0/1838386011 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb068009020 con 0x7fb0580039b0 2024-07-26T23:34:15.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.246+0000 7fb062ffd640 1 -- 172.21.15.139:0/1838386011 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb068022a70 con 0x7fb0580039b0 2024-07-26T23:34:15.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.246+0000 7fb062ffd640 10 client.4104.objecter ms_dispatch 0x7fb06c000b90 osd_map(1..1 src has 1..1) 2024-07-26T23:34:15.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.246+0000 7fb062ffd640 3 client.4104.objecter handle_osd_map got epochs [1,1] > 0 2024-07-26T23:34:15.248 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.246+0000 7fb062ffd640 3 client.4104.objecter handle_osd_map decoding full epoch 1 2024-07-26T23:34:15.248 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.246+0000 7fb062ffd640 20 client.4104.objecter dump_active .. 0 homeless 2024-07-26T23:34:15.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.397+0000 7fb062ffd640 1 -- 172.21.15.139:0/1838386011 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7fb068013040 con 0x7fb0580039b0 2024-07-26T23:34:15.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.398+0000 7fb062ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:15.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.398+0000 7fb062ffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:15.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.471+0000 7fb074668640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-07-26T23:34:15.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.471+0000 7fb074668640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-07-26T23:34:15.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.471+0000 7fb074668640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:15.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.471+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb06c0b8cb0 con 0x7fb0580039b0 2024-07-26T23:34:15.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.645+0000 7fb062ffd640 1 -- 172.21.15.139:0/1838386011 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb06800bd60 con 0x7fb0580039b0 2024-07-26T23:34:15.648 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.646+0000 7fb062ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-07-26T23:34:15.648 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.646+0000 7fb062ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-07-26T23:34:15.648 INFO:teuthology.orchestra.run.smithi139.stderr:adjusted tunables profile to default 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 10 librados: watch_flush enter 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 10 librados: watch_flush exit 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 20 client.4104.objecter shutdown clearing up homeless session... 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 10 client.4104.objecter successfully canceled tick 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 10 monclient: shutdown 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0580039b0 msgr2=0x7fb06c141940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:15.650 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0580039b0 0x7fb06c141940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb068018d90 tx=0x7fb06800c7f0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 shutdown_connections 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb058005290 0x7fb06c1d93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb058004800 0x7fb06c1d8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 --2- 172.21.15.139:0/1838386011 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0580039b0 0x7fb06c141940 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 >> 172.21.15.139:0/1838386011 conn(0x7fb06c0b62c0 msgr2=0x7fb06c0b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 shutdown_connections 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 -- 172.21.15.139:0/1838386011 wait complete. 2024-07-26T23:34:15.651 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:15.647+0000 7fb074668640 1 librados: shutdown 2024-07-26T23:34:15.660 INFO:tasks.ceph:check_enable_crimson: False 2024-07-26T23:34:15.660 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-07-26T23:34:15.661 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:34:15.661 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-07-26T23:34:15.687 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:34:15.687 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-07-26T23:34:15.754 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:34:15.754 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-07-26T23:34:15.820 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:34:15.820 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-07-26T23:34:15.886 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:34:15.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-07-26T23:34:15.911 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:34:15.911 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-07-26T23:34:15.976 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:34:15.976 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-07-26T23:34:16.041 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-07-26T23:34:16.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-07-26T23:34:16.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 26f37aa0-18a9-4f0a-9083-5cfb9a309297 0 2024-07-26T23:34:16.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.246+0000 7f52499ec640 10 monclient: get_monmap_and_config 2024-07-26T23:34:16.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.246+0000 7f52499ec640 10 monclient: build_initial_monmap 2024-07-26T23:34:16.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.246+0000 7f52499ec640 10 monclient: monmap: 2024-07-26T23:34:16.247 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:16.247 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.247 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:16.246946+0000 2024-07-26T23:34:16.247 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:16.246946+0000 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:16.248 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.246+0000 7f52499ec640 1 Processor -- start 2024-07-26T23:34:16.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.246+0000 7f52499ec640 1 -- start start 2024-07-26T23:34:16.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.246+0000 7f52499ec640 10 monclient: init 2024-07-26T23:34:16.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:16.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-07-26T23:34:16.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f52441470c0 0x7f5244140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient(hunting): picked mon.noname-a con 0x7f52441470c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:16.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5244140e90 0x7f52440b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient(hunting): picked mon.noname-c con 0x7f5244140e90 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52440b4700 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5244147ce0 con 0x7f52440b4700 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f52440acaf0 con 0x7f52441470c0 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f52441477c0 con 0x7f5244140e90 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.247+0000 7f52499ec640 10 monclient(hunting): authenticate will time out at 1280.277517s 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f523bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5244140e90 0x7f52440b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f5242ffd640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f52441470c0 0x7f5244140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f523bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5244140e90 0x7f52440b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43396/0 (socket says 172.21.15.177:43396) 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f523bfff640 1 -- 172.21.15.177:0/926438701 learned_addr learned my addr 172.21.15.177:0/926438701 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f523bfff640 10 monclient(hunting): get_auth_request con 0x7f5244140e90 auth_method 0 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f523bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f523bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f523bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f5242ffd640 10 monclient(hunting): get_auth_request con 0x7f52441470c0 auth_method 0 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f5242ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f5242ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.248+0000 7f5242ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.249+0000 7f523bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.249+0000 7f523bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.249+0000 7f523bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.249+0000 7f5242ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.249+0000 7f5242ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.249+0000 7f5242ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-07-26T23:34:16.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 1 -- 172.21.15.177:0/926438701 >> v2:172.21.15.139:3300/0 conn(0x7f52441470c0 msgr2=0x7f5244140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:16.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 1 --2- 172.21.15.177:0/926438701 >> v2:172.21.15.139:3300/0 conn(0x7f52441470c0 0x7f5244140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 1 -- 172.21.15.177:0/926438701 >> v1:172.21.15.139:6789/0 conn(0x7f52440b4700 legacy=0x7f52441438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:16.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 1 monclient: found mon.noname-c 2024-07-26T23:34:16.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:16.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:34:16.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.250+0000 7f523bfff640 1 -- 172.21.15.177:0/926438701 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52441474a0 con 0x7f5244140e90 2024-07-26T23:34:16.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.253+0000 7f523bfff640 1 --2- 172.21.15.177:0/926438701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52440b41c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f52440ac100 tx=0x7f522c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=956ee0a4ef03c385 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:16.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 1 -- 172.21.15.177:0/926438701 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c035020 con 0x7f5244140e90 2024-07-26T23:34:16.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: dump: 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.285 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: _finish_auth 0 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: _check_auth_tickets 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 1 -- 172.21.15.177:0/926438701 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522c031870 con 0x7f5244140e90 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 1 -- 172.21.15.177:0/926438701 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c031c40 con 0x7f5244140e90 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52499ec640 5 monclient: authenticate success, global_id 4105 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52499ec640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52499ec640 4 set_mon_vals no callback set 2024-07-26T23:34:16.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:16.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52427fc640 10 monclient: dump: 2024-07-26T23:34:16.287 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.288 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52499ec640 10 monclient: shutdown 2024-07-26T23:34:16.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52499ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52499ec640 1 -- 172.21.15.177:0/926438701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 msgr2=0x7f52440b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.283+0000 7f52499ec640 1 --2- 172.21.15.177:0/926438701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52440b41c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f52440ac100 tx=0x7f522c02b910 comp rx=0 tx=0).stop 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 1 -- 172.21.15.177:0/926438701 shutdown_connections 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 1 --2- 172.21.15.177:0/926438701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52440b41c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 1 --2- 172.21.15.177:0/926438701 >> v2:172.21.15.139:3300/0 conn(0x7f52441470c0 0x7f5244140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 1 -- 172.21.15.177:0/926438701 >> 172.21.15.177:0/926438701 conn(0x7f52440b62c0 msgr2=0x7f52440b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 1 -- 172.21.15.177:0/926438701 shutdown_connections 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 1 -- 172.21.15.177:0/926438701 wait complete. 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 10 monclient: build_initial_monmap 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.284+0000 7f52499ec640 10 monclient: monmap: 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:16.292 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:16.293 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:16.293 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:16.293 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:16.293 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.293 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:16.293 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:16.293 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 librados: starting msgr at 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 librados: starting objecter 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 Processor -- start 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 -- start start 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 librados: setting wanted keys 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 librados: calling monclient init 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient: init 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f52440b4700 0x7f52441cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): picked mon.noname-c con 0x7f52440b4700 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:16.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52441d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): picked mon.noname-b con 0x7f5244140e90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52441470c0 0x7f52440af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): picked mon.noname-a con 0x7f52441470c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5244141410 con 0x7f52441470c0 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5244074850 con 0x7f5244140e90 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f52440b4b10 con 0x7f52440b4700 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.285+0000 7f52499ec640 10 monclient(hunting): authenticate will time out at 1280.315926s 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.286+0000 7f523bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52441d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.286+0000 7f523bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52441d8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43400/0 (socket says 172.21.15.177:43400) 2024-07-26T23:34:16.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 1 -- 172.21.15.177:0/1255372868 learned_addr learned my addr 172.21.15.177:0/1255372868 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:16.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 10 monclient(hunting): get_auth_request con 0x7f5244140e90 auth_method 0 2024-07-26T23:34:16.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.287+0000 7f523bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.306 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 1 -- 172.21.15.177:0/1255372868 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f52440b4700 msgr2=0x7f52441cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 1 --2- 172.21.15.177:0/1255372868 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f52440b4700 0x7f52441cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 1 -- 172.21.15.177:0/1255372868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52441470c0 msgr2=0x7f52440af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 1 --2- 172.21.15.177:0/1255372868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52441470c0 0x7f52440af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 1 monclient: found mon.noname-b 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:16.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:34:16.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.304+0000 7f523bfff640 1 -- 172.21.15.177:0/1255372868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52440b0a70 con 0x7f5244140e90 2024-07-26T23:34:16.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.307+0000 7f523bfff640 1 --2- 172.21.15.177:0/1255372868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52441d8c70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f522c000c00 tx=0x7f522c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:16.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.307+0000 7f5240ff9640 10 client.?.objecter ms_handle_connect 0x7f5244140e90 2024-07-26T23:34:16.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.307+0000 7f5240ff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:16.314 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 1 -- 172.21.15.177:0/1255372868 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c040070 con 0x7f5244140e90 2024-07-26T23:34:16.314 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.314 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: dump: 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.315 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: _finish_auth 0 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: _check_auth_tickets 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 1 -- 172.21.15.177:0/1255372868 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522c030d00 con 0x7f5244140e90 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 1 -- 172.21.15.177:0/1255372868 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c035040 con 0x7f5244140e90 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5240ff9640 10 monclient: dump: 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.316 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 5 monclient: authenticate success, global_id 4108 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 10 monclient: _renew_subs 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52440aed00 con 0x7f5244140e90 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f5241ffb640 4 set_mon_vals no callback set 2024-07-26T23:34:16.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:16.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 10 monclient: _renew_subs 2024-07-26T23:34:16.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:16.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52441d67b0 con 0x7f5244140e90 2024-07-26T23:34:16.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.312+0000 7f52499ec640 1 librados: init done 2024-07-26T23:34:16.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.314+0000 7f52499ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:16.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.314+0000 7f52499ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:16.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.314+0000 7f52499ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:16.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.314+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52440b41c0 con 0x7f5244140e90 2024-07-26T23:34:16.338 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.337+0000 7f5240ff9640 1 -- 172.21.15.177:0/1255372868 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f522c04b060 con 0x7f5244140e90 2024-07-26T23:34:16.338 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.337+0000 7f5240ff9640 1 -- 172.21.15.177:0/1255372868 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f522c04d520 con 0x7f5244140e90 2024-07-26T23:34:16.338 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.337+0000 7f5240ff9640 10 client.4108.objecter ms_dispatch 0x7f5244000b90 osd_map(2..2 src has 1..2) 2024-07-26T23:34:16.339 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.337+0000 7f5240ff9640 3 client.4108.objecter handle_osd_map got epochs [2,2] > 0 2024-07-26T23:34:16.339 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.337+0000 7f5240ff9640 3 client.4108.objecter handle_osd_map decoding full epoch 2 2024-07-26T23:34:16.339 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.337+0000 7f5240ff9640 20 client.4108.objecter dump_active .. 0 homeless 2024-07-26T23:34:16.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.507+0000 7f5240ff9640 1 -- 172.21.15.177:0/1255372868 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7f522c03a9f0 con 0x7f5244140e90 2024-07-26T23:34:16.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.507+0000 7f5240ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:16.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.507+0000 7f5240ff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:16.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.580+0000 7f52499ec640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26f37aa0-18a9-4f0a-9083-5cfb9a309297", "id": 0}] 2024-07-26T23:34:16.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.580+0000 7f52499ec640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26f37aa0-18a9-4f0a-9083-5cfb9a309297", "id": 0}] 2024-07-26T23:34:16.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.580+0000 7f52499ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:16.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.580+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "26f37aa0-18a9-4f0a-9083-5cfb9a309297", "id": 0} v 0) -- 0x7f52440aef70 con 0x7f5244140e90 2024-07-26T23:34:16.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.759+0000 7f5240ff9640 1 -- 172.21.15.177:0/1255372868 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26f37aa0-18a9-4f0a-9083-5cfb9a309297", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f522c03abd0 con 0x7f5244140e90 2024-07-26T23:34:16.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.759+0000 7f5240ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26f37aa0-18a9-4f0a-9083-5cfb9a309297", "id": 0}] 2024-07-26T23:34:16.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.759+0000 7f5240ff9640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:16.761 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-07-26T23:34:16.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 10 librados: watch_flush enter 2024-07-26T23:34:16.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 10 librados: watch_flush exit 2024-07-26T23:34:16.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 20 client.4108.objecter shutdown clearing up homeless session... 2024-07-26T23:34:16.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 10 client.4108.objecter successfully canceled tick 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 10 monclient: shutdown 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 msgr2=0x7f52441d8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 --2- 172.21.15.177:0/1255372868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52441d8c70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f522c000c00 tx=0x7f522c0046c0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 shutdown_connections 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 --2- 172.21.15.177:0/1255372868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f52441470c0 0x7f52440af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 --2- 172.21.15.177:0/1255372868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5244140e90 0x7f52441d8c70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 --2- 172.21.15.177:0/1255372868 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f52440b4700 0x7f52441cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 >> 172.21.15.177:0/1255372868 conn(0x7f52440b62c0 msgr2=0x7f52440b8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 shutdown_connections 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 -- 172.21.15.177:0/1255372868 wait complete. 2024-07-26T23:34:16.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.760+0000 7f52499ec640 1 librados: shutdown 2024-07-26T23:34:16.772 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 2f5fd361-2a42-47ec-b4ad-b2636b9db59c 1 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.871+0000 7f46f0657640 10 monclient: get_monmap_and_config 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.871+0000 7f46f0657640 10 monclient: build_initial_monmap 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.871+0000 7f46f0657640 10 monclient: monmap: 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:16.872315+0000 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:16.872315+0000 2024-07-26T23:34:16.873 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 Processor -- start 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 -- start start 2024-07-26T23:34:16.874 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient: init 2024-07-26T23:34:16.876 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:16.876 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-07-26T23:34:16.876 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e8140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.876 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): picked mon.noname-b con 0x7f46e81470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:16.876 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f46e8140e90 0x7f46e80b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.876 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): picked mon.noname-a con 0x7f46e8140e90 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f46e80b4700 0x7f46e81438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): picked mon.noname-c con 0x7f46e80b4700 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f46e8147ce0 con 0x7f46e8140e90 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f46e80acaf0 con 0x7f46e80b4700 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f46e81477c0 con 0x7f46e81470c0 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.872+0000 7f46f0657640 10 monclient(hunting): authenticate will time out at 1280.902938s 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46ee3f3640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e8140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.877 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46edbf2640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f46e8140e90 0x7f46e80b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46ee3f3640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e8140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:50578/0 (socket says 172.21.15.177:50578) 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46ee3f3640 1 -- 172.21.15.177:0/385998639 learned_addr learned my addr 172.21.15.177:0/385998639 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46ee3f3640 10 monclient(hunting): get_auth_request con 0x7f46e81470c0 auth_method 0 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46ee3f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46ee3f3640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46ee3f3640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46edbf2640 10 monclient(hunting): get_auth_request con 0x7f46e8140e90 auth_method 0 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46edbf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46edbf2640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.873+0000 7f46edbf2640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.874+0000 7f46ee3f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.874+0000 7f46ee3f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.878 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.874+0000 7f46ee3f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.879 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.874+0000 7f46edbf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.879 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.874+0000 7f46edbf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.879 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.874+0000 7f46edbf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.879 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46eebf4640 1 --2- 172.21.15.177:0/385998639 >> v2:172.21.15.177:3300/0 conn(0x7f46e80b4700 0x7f46e81438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 1 -- 172.21.15.177:0/385998639 >> v2:172.21.15.177:3300/0 conn(0x7f46e80b4700 msgr2=0x7f46e81438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 1 --2- 172.21.15.177:0/385998639 >> v2:172.21.15.177:3300/0 conn(0x7f46e80b4700 0x7f46e81438d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 1 -- 172.21.15.177:0/385998639 >> v2:172.21.15.139:3300/0 conn(0x7f46e8140e90 msgr2=0x7f46e80b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 1 --2- 172.21.15.177:0/385998639 >> v2:172.21.15.139:3300/0 conn(0x7f46e8140e90 0x7f46e80b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 10 monclient: _finish_hunting 0 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 1 monclient: found mon.noname-b 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.875+0000 7f46ee3f3640 1 -- 172.21.15.177:0/385998639 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e80b4b90 con 0x7f46e81470c0 2024-07-26T23:34:16.880 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.878+0000 7f46ee3f3640 1 --2- 172.21.15.177:0/385998639 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e8140950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f46e80b7100 tx=0x7f46e402b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=550a0ca6b9627b1a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:16.884 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 1 -- 172.21.15.177:0/385998639 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e402e070 con 0x7f46e81470c0 2024-07-26T23:34:16.884 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: dump: 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.885 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: _finish_auth 0 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: _check_auth_tickets 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 1 -- 172.21.15.177:0/385998639 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e4037070 con 0x7f46e81470c0 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46f0657640 5 monclient: authenticate success, global_id 4109 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 1 -- 172.21.15.177:0/385998639 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4031a00 con 0x7f46e81470c0 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46f0657640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:16.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46f0657640 4 set_mon_vals no callback set 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.881+0000 7f46ed3f1640 10 monclient: dump: 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.887 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.888 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 10 monclient: shutdown 2024-07-26T23:34:16.888 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:16.888 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 -- 172.21.15.177:0/385998639 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 msgr2=0x7f46e8140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.888 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 --2- 172.21.15.177:0/385998639 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e8140950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f46e80b7100 tx=0x7f46e402b880 comp rx=0 tx=0).stop 2024-07-26T23:34:16.888 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 -- 172.21.15.177:0/385998639 shutdown_connections 2024-07-26T23:34:16.888 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 --2- 172.21.15.177:0/385998639 >> v2:172.21.15.177:3300/0 conn(0x7f46e80b4700 0x7f46e81438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.888 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 --2- 172.21.15.177:0/385998639 >> v2:172.21.15.139:3300/0 conn(0x7f46e8140e90 0x7f46e80b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 --2- 172.21.15.177:0/385998639 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e8140950 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 -- 172.21.15.177:0/385998639 >> 172.21.15.177:0/385998639 conn(0x7f46e80b62c0 msgr2=0x7f46e80b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 -- 172.21.15.177:0/385998639 shutdown_connections 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.882+0000 7f46f0657640 1 -- 172.21.15.177:0/385998639 wait complete. 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.883+0000 7f46f0657640 10 monclient: build_initial_monmap 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.883+0000 7f46f0657640 10 monclient: monmap: 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.889 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:16.890 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:16.890 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:16.890 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.883+0000 7f46f0657640 1 librados: starting msgr at 2024-07-26T23:34:16.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.883+0000 7f46f0657640 1 librados: starting objecter 2024-07-26T23:34:16.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.883+0000 7f46f0657640 1 Processor -- start 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 -- start start 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 librados: setting wanted keys 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 librados: calling monclient init 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient: init 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46e80b4700 0x7f46e813b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): picked mon.noname-b con 0x7f46e80b4700 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46e8140e90 0x7f46e8143b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): picked mon.noname-a con 0x7f46e8140e90 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e81416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:16.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): picked mon.noname-c con 0x7f46e81470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f46e8074850 con 0x7f46e8140e90 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46e80699f0 con 0x7f46e80b4700 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f46e8141410 con 0x7f46e81470c0 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.884+0000 7f46f0657640 10 monclient(hunting): authenticate will time out at 1280.914586s 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46ee3f3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46e80b4700 0x7f46e813b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.896 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e81416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46e8140e90 0x7f46e8143b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e81416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:50594/0 (socket says 172.21.15.177:50594) 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 1 -- 172.21.15.177:0/403708507 learned_addr learned my addr 172.21.15.177:0/403708507 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 10 monclient(hunting): get_auth_request con 0x7f46e81470c0 auth_method 0 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 10 monclient(hunting): get_auth_request con 0x7f46e8140e90 auth_method 0 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46ee3f3640 10 monclient(hunting): get_auth_request con 0x7f46e80b4700 auth_method 0 2024-07-26T23:34:16.898 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46ee3f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:16.898 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46ee3f3640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:16.898 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46ee3f3640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:16.898 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.898 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.898 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46eebf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.898 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.885+0000 7f46edbf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.886+0000 7f46ee3f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.886+0000 7f46ee3f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.886+0000 7f46ee3f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 10 monclient(hunting): handle_auth_done global_id 4119 payload 824 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 1 -- 172.21.15.177:0/403708507 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 msgr2=0x7f46e81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 1 --2- 172.21.15.177:0/403708507 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e81416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 1 -- 172.21.15.177:0/403708507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46e80b4700 msgr2=0x7f46e813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 1 --2- 172.21.15.177:0/403708507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46e80b4700 0x7f46e813b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 10 monclient: _finish_hunting 0 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 1 monclient: found mon.noname-a 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.888+0000 7f46edbf2640 1 -- 172.21.15.177:0/403708507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e8137720 con 0x7f46e8140e90 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.891+0000 7f46edbf2640 1 --2- 172.21.15.177:0/403708507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46e8140e90 0x7f46e8143b20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46d800e820 tx=0x7f46d800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.891+0000 7f46deffd640 10 client.?.objecter ms_handle_connect 0x7f46e8140e90 2024-07-26T23:34:16.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.891+0000 7f46deffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:16.901 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 1 -- 172.21.15.177:0/403708507 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8017070 con 0x7f46e8140e90 2024-07-26T23:34:16.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:34:16.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: dump: 2024-07-26T23:34:16.902 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.902 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.902 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: _finish_auth 0 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 1 -- 172.21.15.177:0/403708507 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d800c940 con 0x7f46e8140e90 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:16.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 1 -- 172.21.15.177:0/403708507 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d800cd10 con 0x7f46e8140e90 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 5 monclient: authenticate success, global_id 4119 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 10 monclient: _renew_subs 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e8137a00 con 0x7f46e8140e90 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46ecbf0640 4 set_mon_vals no callback set 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46deffd640 10 monclient: dump: 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:16.904 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 10 monclient: _renew_subs 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e8141c10 con 0x7f46e8140e90 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.895+0000 7f46f0657640 1 librados: init done 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.896+0000 7f46f0657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.896+0000 7f46f0657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.896+0000 7f46f0657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:16.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:16.896+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e8140c80 con 0x7f46e8140e90 2024-07-26T23:34:17.001 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.000+0000 7f46deffd640 1 -- 172.21.15.177:0/403708507 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f46d801a070 con 0x7f46e8140e90 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.004+0000 7f46deffd640 1 -- 172.21.15.177:0/403708507 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f46d8013470 con 0x7f46e8140e90 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.004+0000 7f46deffd640 10 client.4119.objecter ms_dispatch 0x7f46e8000b90 osd_map(3..3 src has 1..3) 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.004+0000 7f46deffd640 3 client.4119.objecter handle_osd_map got epochs [3,3] > 0 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.004+0000 7f46deffd640 3 client.4119.objecter handle_osd_map decoding full epoch 3 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.005+0000 7f46deffd640 20 client.4119.objecter dump_active .. 0 homeless 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.005+0000 7f46deffd640 1 -- 172.21.15.177:0/403708507 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7f46e8140c80 con 0x7f46e8140e90 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.005+0000 7f46deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:17.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.005+0000 7f46deffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:17.080 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.079+0000 7f46f0657640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2f5fd361-2a42-47ec-b4ad-b2636b9db59c", "id": 1}] 2024-07-26T23:34:17.080 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.079+0000 7f46f0657640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2f5fd361-2a42-47ec-b4ad-b2636b9db59c", "id": 1}] 2024-07-26T23:34:17.080 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.079+0000 7f46f0657640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:17.080 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.079+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2f5fd361-2a42-47ec-b4ad-b2636b9db59c", "id": 1} v 0) -- 0x7f46e80b41c0 con 0x7f46e8140e90 2024-07-26T23:34:17.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.120+0000 7f46deffd640 1 -- 172.21.15.177:0/403708507 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2f5fd361-2a42-47ec-b4ad-b2636b9db59c", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f46d8005990 con 0x7f46e8140e90 2024-07-26T23:34:17.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.120+0000 7f46deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2f5fd361-2a42-47ec-b4ad-b2636b9db59c", "id": 1}] 2024-07-26T23:34:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.120+0000 7f46deffd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:17.122 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-07-26T23:34:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 10 librados: watch_flush enter 2024-07-26T23:34:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 10 librados: watch_flush exit 2024-07-26T23:34:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 20 client.4119.objecter shutdown clearing up homeless session... 2024-07-26T23:34:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 10 client.4119.objecter successfully canceled tick 2024-07-26T23:34:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 10 monclient: shutdown 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46e8140e90 msgr2=0x7f46e8143b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 --2- 172.21.15.177:0/403708507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46e8140e90 0x7f46e8143b20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46d800e820 tx=0x7f46d800ece0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 shutdown_connections 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 --2- 172.21.15.177:0/403708507 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f46e81470c0 0x7f46e81416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 --2- 172.21.15.177:0/403708507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46e8140e90 0x7f46e8143b20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 --2- 172.21.15.177:0/403708507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46e80b4700 0x7f46e813b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 >> 172.21.15.177:0/403708507 conn(0x7f46e80b62c0 msgr2=0x7f46e80691f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 shutdown_connections 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 -- 172.21.15.177:0/403708507 wait complete. 2024-07-26T23:34:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.121+0000 7f46f0657640 1 librados: shutdown 2024-07-26T23:34:17.133 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new ea342704-f04a-4db8-8519-ed7802ae361f 2 2024-07-26T23:34:17.232 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.231+0000 7f4ef4980640 10 monclient: get_monmap_and_config 2024-07-26T23:34:17.232 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.231+0000 7f4ef4980640 10 monclient: build_initial_monmap 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.231+0000 7f4ef4980640 10 monclient: monmap: 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:17.232447+0000 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:17.232447+0000 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:17.233 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 1 Processor -- start 2024-07-26T23:34:17.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 1 -- start start 2024-07-26T23:34:17.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient: init 2024-07-26T23:34:17.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ef01470c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f4ef00b41c0 0x7f4ef00b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef00b41c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef014c1d0 0x7f4ef014e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef014c1d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef0147ce0 con 0x7f4ef01470c0 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef00acaf0 con 0x7f4ef00b41c0 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef01477c0 con 0x7f4ef014c1d0 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:17.237 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.232+0000 7f4ef4980640 10 monclient(hunting): authenticate will time out at 1281.262850s 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4eeed9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef014c1d0 0x7f4ef014e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4ee5d9b640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f4ef00b41c0 0x7f4ef00b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4eee59c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f4ef01470c0 0x7f4ef0140950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:41216/0 (socket says 172.21.15.177:41216) 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4eee59c640 1 -- 172.21.15.177:0/77016790 learned_addr learned my addr 172.21.15.177:0/77016790 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4eeed9d640 10 monclient(hunting): get_auth_request con 0x7f4ef014c1d0 auth_method 0 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4eeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4eeed9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4eeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4ee5d9b640 10 monclient(hunting): get_auth_request con 0x7f4ef00b41c0 auth_method 0 2024-07-26T23:34:17.238 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4ee5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4ee5d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.233+0000 7f4ee5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.234+0000 7f4eeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.234+0000 7f4eeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.234+0000 7f4eeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.234+0000 7f4ee5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.234+0000 7f4ee5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.239 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.234+0000 7f4ee5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 1 -- 172.21.15.177:0/77016790 >> v2:172.21.15.139:3300/0 conn(0x7f4ef00b41c0 msgr2=0x7f4ef00b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 1 --2- 172.21.15.177:0/77016790 >> v2:172.21.15.139:3300/0 conn(0x7f4ef00b41c0 0x7f4ef00b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 1 -- 172.21.15.177:0/77016790 >> v1:172.21.15.177:6789/0 conn(0x7f4ef01470c0 legacy=0x7f4ef0140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 10 monclient: _finish_hunting 0 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 1 monclient: found mon.noname-b 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.240 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.235+0000 7f4eeed9d640 1 -- 172.21.15.177:0/77016790 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef01474a0 con 0x7f4ef014c1d0 2024-07-26T23:34:17.241 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.237+0000 7f4eeed9d640 1 --2- 172.21.15.177:0/77016790 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef014c1d0 0x7f4ef014e5c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009d90 tx=0x7f4ee802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f9402d4de3d3935 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 1 -- 172.21.15.177:0/77016790 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8039070 con 0x7f4ef014c1d0 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: dump: 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.242 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: _finish_auth 0 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: _check_auth_tickets 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 1 -- 172.21.15.177:0/77016790 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee802cbc0 con 0x7f4ef014c1d0 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 1 -- 172.21.15.177:0/77016790 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8034440 con 0x7f4ef014c1d0 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 5 monclient: authenticate success, global_id 4115 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:17.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 4 set_mon_vals no callback set 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4eedd9b640 10 monclient: dump: 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.244 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 10 monclient: shutdown 2024-07-26T23:34:17.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:17.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 -- 172.21.15.177:0/77016790 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef014c1d0 msgr2=0x7f4ef014e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 --2- 172.21.15.177:0/77016790 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef014c1d0 0x7f4ef014e5c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009d90 tx=0x7f4ee802be10 comp rx=0 tx=0).stop 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 -- 172.21.15.177:0/77016790 shutdown_connections 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 --2- 172.21.15.177:0/77016790 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef014c1d0 0x7f4ef014e5c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 --2- 172.21.15.177:0/77016790 >> v2:172.21.15.139:3300/0 conn(0x7f4ef00b41c0 0x7f4ef00b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 -- 172.21.15.177:0/77016790 >> 172.21.15.177:0/77016790 conn(0x7f4ef00b62c0 msgr2=0x7f4ef00b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 -- 172.21.15.177:0/77016790 shutdown_connections 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.241+0000 7f4ef4980640 1 -- 172.21.15.177:0/77016790 wait complete. 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.242+0000 7f4ef4980640 10 monclient: build_initial_monmap 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.242+0000 7f4ef4980640 10 monclient: monmap: 2024-07-26T23:34:17.249 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.242+0000 7f4ef4980640 1 librados: starting msgr at 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.242+0000 7f4ef4980640 1 librados: starting objecter 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 Processor -- start 2024-07-26T23:34:17.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 -- start start 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 librados: setting wanted keys 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 librados: calling monclient init 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient: init 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef00b41c0 0x7f4ef0148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef00b41c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef01470c0 0x7f4ef0143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef01470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef014c1d0 0x7f4ef0144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef014c1d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef0141410 con 0x7f4ef00b41c0 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef0074850 con 0x7f4ef014c1d0 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef00b4b20 con 0x7f4ef01470c0 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.243+0000 7f4ef4980640 10 monclient(hunting): authenticate will time out at 1281.273898s 2024-07-26T23:34:17.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4ee5d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef01470c0 0x7f4ef0143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4eeed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef014c1d0 0x7f4ef0144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4eee59c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef00b41c0 0x7f4ef0148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4ee5d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef01470c0 0x7f4ef0143d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:50612/0 (socket says 172.21.15.177:50612) 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4ee5d9b640 1 -- 172.21.15.177:0/3959750907 learned_addr learned my addr 172.21.15.177:0/3959750907 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4ee5d9b640 10 monclient(hunting): get_auth_request con 0x7f4ef01470c0 auth_method 0 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4ee5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4ee5d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4ee5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4eee59c640 10 monclient(hunting): get_auth_request con 0x7f4ef00b41c0 auth_method 0 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4eee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4eee59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.244+0000 7f4eee59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eeed9d640 10 monclient(hunting): get_auth_request con 0x7f4ef014c1d0 auth_method 0 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eeed9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4ee5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4ee5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4ee5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.245+0000 7f4eeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 10 monclient(hunting): handle_auth_done global_id 4118 payload 824 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 1 -- 172.21.15.177:0/3959750907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef014c1d0 msgr2=0x7f4ef0144280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 1 --2- 172.21.15.177:0/3959750907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef014c1d0 0x7f4ef0144280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 1 -- 172.21.15.177:0/3959750907 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef00b41c0 msgr2=0x7f4ef0148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 1 --2- 172.21.15.177:0/3959750907 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef00b41c0 0x7f4ef0148aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 10 monclient: _finish_hunting 0 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 1 monclient: found mon.noname-c 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.248+0000 7f4ee5d9b640 1 -- 172.21.15.177:0/3959750907 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef014af70 con 0x7f4ef01470c0 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.250+0000 7f4ee5d9b640 1 --2- 172.21.15.177:0/3959750907 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef01470c0 0x7f4ef0143d40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800e870 tx=0x7f4ed800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.250+0000 7f4ee77fe640 10 client.?.objecter ms_handle_connect 0x7f4ef01470c0 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.250+0000 7f4ee77fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:17.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 1 -- 172.21.15.177:0/3959750907 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8017070 con 0x7f4ef01470c0 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: dump: 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.258 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: _finish_auth 0 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: _check_auth_tickets 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 1 -- 172.21.15.177:0/3959750907 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed800c940 con 0x7f4ef01470c0 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 5 monclient: authenticate success, global_id 4118 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 10 monclient: _renew_subs 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.259 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef01d6960 con 0x7f4ef01470c0 2024-07-26T23:34:17.260 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:17.260 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 1 -- 172.21.15.177:0/3959750907 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed800cd30 con 0x7f4ef01470c0 2024-07-26T23:34:17.260 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.260 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:17.261 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ee77fe640 10 monclient: dump: 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4eed59a640 4 set_mon_vals no callback set 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:17.262 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 10 monclient: _renew_subs 2024-07-26T23:34:17.263 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.263 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef0147fa0 con 0x7f4ef01470c0 2024-07-26T23:34:17.263 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.254+0000 7f4ef4980640 1 librados: init done 2024-07-26T23:34:17.263 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.255+0000 7f4ef4980640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:17.263 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.255+0000 7f4ef4980640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:17.263 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.255+0000 7f4ef4980640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.263 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.255+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef00b45e0 con 0x7f4ef01470c0 2024-07-26T23:34:17.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.439+0000 7f4ee77fe640 1 -- 172.21.15.177:0/3959750907 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4ed801a070 con 0x7f4ef01470c0 2024-07-26T23:34:17.447 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 1 -- 172.21.15.177:0/3959750907 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f4ed8013470 con 0x7f4ef01470c0 2024-07-26T23:34:17.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 10 client.4118.objecter ms_dispatch 0x7f4ef0000b90 osd_map(4..4 src has 1..4) 2024-07-26T23:34:17.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 3 client.4118.objecter handle_osd_map got epochs [4,4] > 0 2024-07-26T23:34:17.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 3 client.4118.objecter handle_osd_map decoding full epoch 4 2024-07-26T23:34:17.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 20 client.4118.objecter dump_active .. 0 homeless 2024-07-26T23:34:17.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 1 -- 172.21.15.177:0/3959750907 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7f4ed8006490 con 0x7f4ef01470c0 2024-07-26T23:34:17.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:17.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.446+0000 7f4ee77fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:17.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.520+0000 7f4ef4980640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ea342704-f04a-4db8-8519-ed7802ae361f", "id": 2}] 2024-07-26T23:34:17.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.520+0000 7f4ef4980640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ea342704-f04a-4db8-8519-ed7802ae361f", "id": 2}] 2024-07-26T23:34:17.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.520+0000 7f4ef4980640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.520+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ea342704-f04a-4db8-8519-ed7802ae361f", "id": 2} v 0) -- 0x7f4ef013fd50 con 0x7f4ef01470c0 2024-07-26T23:34:17.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.646+0000 7f4ee77fe640 1 -- 172.21.15.177:0/3959750907 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea342704-f04a-4db8-8519-ed7802ae361f", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f4ed8006670 con 0x7f4ef01470c0 2024-07-26T23:34:17.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.646+0000 7f4ee77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ea342704-f04a-4db8-8519-ed7802ae361f", "id": 2}] 2024-07-26T23:34:17.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.646+0000 7f4ee77fe640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:17.648 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 10 librados: watch_flush enter 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 10 librados: watch_flush exit 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 20 client.4118.objecter shutdown clearing up homeless session... 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 10 client.4118.objecter successfully canceled tick 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 10 monclient: shutdown 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef01470c0 msgr2=0x7f4ef0143d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.647+0000 7f4ef4980640 1 --2- 172.21.15.177:0/3959750907 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef01470c0 0x7f4ef0143d40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800e870 tx=0x7f4ed800ed30 comp rx=0 tx=0).stop 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 shutdown_connections 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 --2- 172.21.15.177:0/3959750907 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4ef01470c0 0x7f4ef0143d40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 --2- 172.21.15.177:0/3959750907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef014c1d0 0x7f4ef0144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 --2- 172.21.15.177:0/3959750907 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ef00b41c0 0x7f4ef0148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 >> 172.21.15.177:0/3959750907 conn(0x7f4ef00b62c0 msgr2=0x7f4ef013ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:17.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 shutdown_connections 2024-07-26T23:34:17.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 -- 172.21.15.177:0/3959750907 wait complete. 2024-07-26T23:34:17.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.648+0000 7f4ef4980640 1 librados: shutdown 2024-07-26T23:34:17.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 6f9fd3b7-d9de-4283-aa26-31a80761ebc6 3 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.756+0000 7f8d9efef640 10 monclient: get_monmap_and_config 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.756+0000 7f8d9efef640 10 monclient: build_initial_monmap 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.756+0000 7f8d9efef640 10 monclient: monmap: 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:17.757723+0000 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:17.757723+0000 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:17.758 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.759 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:17.759 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:17.759 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:17.759 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:17.759 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:17.759 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.757+0000 7f8d9efef640 1 Processor -- start 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.757+0000 7f8d9efef640 1 -- start start 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.757+0000 7f8d9efef640 10 monclient: init 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d981470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8d980b41c0 0x7f8d980b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient(hunting): picked mon.noname-c con 0x7f8d980b41c0 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d9814c1d0 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:17.761 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d98147ce0 con 0x7f8d981470c0 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d980acaf0 con 0x7f8d9814c1d0 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8d981477c0 con 0x7f8d980b41c0 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d9efef640 10 monclient(hunting): authenticate will time out at 1281.788348s 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.758+0000 7f8d97fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8d980b41c0 0x7f8d980b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.759+0000 7f8d9cd8b640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f8d981470c0 0x7f8d98140950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.177:58794/0 (socket says 172.21.15.177:58794) 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.759+0000 7f8d9cd8b640 1 -- 172.21.15.177:0/2221076244 learned_addr learned my addr 172.21.15.177:0/2221076244 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.759+0000 7f8d97fff640 10 monclient(hunting): get_auth_request con 0x7f8d980b41c0 auth_method 0 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.759+0000 7f8d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.762 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.759+0000 7f8d97fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.759+0000 7f8d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.760+0000 7f8d97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.760+0000 7f8d97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.764 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.760+0000 7f8d97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.764 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.761+0000 7f8d977fe640 1 -- 172.21.15.177:0/2221076244 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2475140491 0 0) 0x7f8d98147ce0 con 0x7f8d981470c0 2024-07-26T23:34:17.764 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.761+0000 7f8d977fe640 10 monclient(hunting): _negotiate 2024-07-26T23:34:17.764 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.761+0000 7f8d977fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.764 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.761+0000 7f8d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.764 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.761+0000 7f8d977fe640 10 monclient(hunting): my global_id is 4128 2024-07-26T23:34:17.764 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.761+0000 7f8d977fe640 1 -- 172.21.15.177:0/2221076244 --> v1:172.21.15.139:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d88003520 con 0x7f8d981470c0 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 1 -- 172.21.15.177:0/2221076244 >> v1:172.21.15.177:6789/0 conn(0x7f8d9814c1d0 legacy=0x7f8d9814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 1 -- 172.21.15.177:0/2221076244 >> v1:172.21.15.139:6789/0 conn(0x7f8d981470c0 legacy=0x7f8d98140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 1 monclient: found mon.noname-c 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:34:17.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.763+0000 7f8d97fff640 1 -- 172.21.15.177:0/2221076244 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d981474a0 con 0x7f8d980b41c0 2024-07-26T23:34:17.767 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.766+0000 7f8d97fff640 1 --2- 172.21.15.177:0/2221076244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d980b41c0 0x7f8d980b45e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8d840097a0 tx=0x7f8d8402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e6b53c936e6b59a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:17.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 1 -- 172.21.15.177:0/2221076244 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84035020 con 0x7f8d980b41c0 2024-07-26T23:34:17.771 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: dump: 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.772 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: _finish_auth 0 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: _check_auth_tickets 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 1 -- 172.21.15.177:0/2221076244 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d84031830 con 0x7f8d980b41c0 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d9efef640 5 monclient: authenticate success, global_id 4120 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d9efef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d9efef640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:17.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d9efef640 4 set_mon_vals no callback set 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 1 -- 172.21.15.177:0/2221076244 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84031c00 con 0x7f8d980b41c0 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.770+0000 7f8d977fe640 10 monclient: dump: 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.774 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.775 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.775 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.775 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 10 monclient: shutdown 2024-07-26T23:34:17.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:17.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 1 -- 172.21.15.177:0/2221076244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d980b41c0 msgr2=0x7f8d980b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 1 --2- 172.21.15.177:0/2221076244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d980b41c0 0x7f8d980b45e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8d840097a0 tx=0x7f8d8402b8d0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 1 -- 172.21.15.177:0/2221076244 shutdown_connections 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 1 --2- 172.21.15.177:0/2221076244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d980b41c0 0x7f8d980b45e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 1 -- 172.21.15.177:0/2221076244 >> 172.21.15.177:0/2221076244 conn(0x7f8d980b62c0 msgr2=0x7f8d980b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 1 -- 172.21.15.177:0/2221076244 shutdown_connections 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 1 -- 172.21.15.177:0/2221076244 wait complete. 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 10 monclient: build_initial_monmap 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.771+0000 7f8d9efef640 10 monclient: monmap: 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:17.780 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.772+0000 7f8d9efef640 1 librados: starting msgr at 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.772+0000 7f8d9efef640 1 librados: starting objecter 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.772+0000 7f8d9efef640 1 Processor -- start 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.772+0000 7f8d9efef640 1 -- start start 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.772+0000 7f8d9efef640 1 librados: setting wanted keys 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.772+0000 7f8d9efef640 1 librados: calling monclient init 2024-07-26T23:34:17.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.772+0000 7f8d9efef640 10 monclient: init 2024-07-26T23:34:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:34:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d980b41c0 0x7f8d981434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): picked mon.noname-a con 0x7f8d980b41c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8d981470c0 0x7f8d980aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): picked mon.noname-c con 0x7f8d981470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:17.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d9814c1d0 0x7f8d981d9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): picked mon.noname-b con 0x7f8d9814c1d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8d98141410 con 0x7f8d980b41c0 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d98074850 con 0x7f8d9814c1d0 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f8d980b4b20 con 0x7f8d981470c0 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9efef640 10 monclient(hunting): authenticate will time out at 1281.803195s 2024-07-26T23:34:17.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d9cd8b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d980b41c0 0x7f8d981434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.773+0000 7f8d97fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8d981470c0 0x7f8d980aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8d981470c0 0x7f8d980aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:50628/0 (socket says 172.21.15.177:50628) 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 1 -- 172.21.15.177:0/1456245131 learned_addr learned my addr 172.21.15.177:0/1456245131 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 10 monclient(hunting): get_auth_request con 0x7f8d981470c0 auth_method 0 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9d58c640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d9814c1d0 0x7f8d981d9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9cd8b640 10 monclient(hunting): get_auth_request con 0x7f8d980b41c0 auth_method 0 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9cd8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9cd8b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9cd8b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9d58c640 10 monclient(hunting): get_auth_request con 0x7f8d9814c1d0 auth_method 0 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9d58c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9d58c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9d58c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9cd8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9cd8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.774+0000 7f8d9cd8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.775+0000 7f8d9d58c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.775+0000 7f8d9d58c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.775+0000 7f8d9d58c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 10 monclient(hunting): handle_auth_done global_id 4121 payload 824 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 1 -- 172.21.15.177:0/1456245131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d9814c1d0 msgr2=0x7f8d981d9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d9814c1d0 0x7f8d981d9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 1 -- 172.21.15.177:0/1456245131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d980b41c0 msgr2=0x7f8d981434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d980b41c0 0x7f8d981434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:17.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 1 monclient: found mon.noname-c 2024-07-26T23:34:17.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:17.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.777+0000 7f8d97fff640 1 -- 172.21.15.177:0/1456245131 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d980b0670 con 0x7f8d981470c0 2024-07-26T23:34:17.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.779+0000 7f8d97fff640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8d981470c0 0x7f8d980aef50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d84034040 tx=0x7f8d840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.780+0000 7f8d957fa640 10 client.?.objecter ms_handle_connect 0x7f8d981470c0 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.780+0000 7f8d957fa640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 1 -- 172.21.15.177:0/1456245131 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84040070 con 0x7f8d981470c0 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: dump: 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.789 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: _finish_auth 0 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: _check_auth_tickets 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 5 monclient: authenticate success, global_id 4121 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 10 monclient: _renew_subs 2024-07-26T23:34:17.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d980af490 con 0x7f8d981470c0 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 10 monclient: _renew_subs 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d981d9660 con 0x7f8d981470c0 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d9efef640 1 librados: init done 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 1 -- 172.21.15.177:0/1456245131 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d84031800 con 0x7f8d981470c0 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:17.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 1 -- 172.21.15.177:0/1456245131 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84035040 con 0x7f8d981470c0 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d957fa640 10 monclient: dump: 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:17.792 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:17.793 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:17.793 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:17.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.783+0000 7f8d96ffd640 4 set_mon_vals no callback set 2024-07-26T23:34:17.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.784+0000 7f8d9efef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:17.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.784+0000 7f8d9efef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:17.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.784+0000 7f8d9efef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.784+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d78005080 con 0x7f8d981470c0 2024-07-26T23:34:17.881 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.880+0000 7f8d957fa640 1 -- 172.21.15.177:0/1456245131 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d84038020 con 0x7f8d981470c0 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 1 -- 172.21.15.177:0/1456245131 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f8d84044d70 con 0x7f8d981470c0 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 10 client.4121.objecter ms_dispatch 0x7f8d98000b90 osd_map(5..5 src has 1..5) 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 3 client.4121.objecter handle_osd_map got epochs [5,5] > 0 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 3 client.4121.objecter handle_osd_map decoding full epoch 5 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 20 client.4121.objecter dump_active .. 0 homeless 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 1 -- 172.21.15.177:0/1456245131 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7f8d84044410 con 0x7f8d981470c0 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:17.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.884+0000 7f8d957fa640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:17.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.958+0000 7f8d9efef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6f9fd3b7-d9de-4283-aa26-31a80761ebc6", "id": 3}] 2024-07-26T23:34:17.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.958+0000 7f8d9efef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6f9fd3b7-d9de-4283-aa26-31a80761ebc6", "id": 3}] 2024-07-26T23:34:17.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.958+0000 7f8d9efef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:17.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:17.958+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6f9fd3b7-d9de-4283-aa26-31a80761ebc6", "id": 3} v 0) -- 0x7f8d78005640 con 0x7f8d981470c0 2024-07-26T23:34:18.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.042+0000 7f8d957fa640 1 -- 172.21.15.177:0/1456245131 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f9fd3b7-d9de-4283-aa26-31a80761ebc6", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f8d840445f0 con 0x7f8d981470c0 2024-07-26T23:34:18.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.042+0000 7f8d957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6f9fd3b7-d9de-4283-aa26-31a80761ebc6", "id": 3}] 2024-07-26T23:34:18.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.042+0000 7f8d957fa640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:18.044 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 10 librados: watch_flush enter 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 10 librados: watch_flush exit 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 20 client.4121.objecter shutdown clearing up homeless session... 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 10 client.4121.objecter successfully canceled tick 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 10 monclient: shutdown 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8d981470c0 msgr2=0x7f8d980aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8d981470c0 0x7f8d980aef50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d84034040 tx=0x7f8d840046c0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 shutdown_connections 2024-07-26T23:34:18.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8d981470c0 0x7f8d980aef50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d9814c1d0 0x7f8d981d9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 1 --2- 172.21.15.177:0/1456245131 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d980b41c0 0x7f8d981434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.043+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 >> 172.21.15.177:0/1456245131 conn(0x7f8d980b62c0 msgr2=0x7f8d980b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:18.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.044+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 shutdown_connections 2024-07-26T23:34:18.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.044+0000 7f8d9efef640 1 -- 172.21.15.177:0/1456245131 wait complete. 2024-07-26T23:34:18.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.044+0000 7f8d9efef640 1 librados: shutdown 2024-07-26T23:34:18.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 5a61bbd3-3522-461c-bfd5-ac5d0d6103ca 4 2024-07-26T23:34:18.154 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.153+0000 7fa50935b640 10 monclient: get_monmap_and_config 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.153+0000 7fa50935b640 10 monclient: build_initial_monmap 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.153+0000 7fa50935b640 10 monclient: monmap: 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:18.154499+0000 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:18.154499+0000 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:18.155 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:18.156 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:18.156 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:18.156 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:18.156 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.158 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 1 Processor -- start 2024-07-26T23:34:18.158 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 1 -- start start 2024-07-26T23:34:18.158 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient: init 2024-07-26T23:34:18.158 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5040b4c10 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa504074810 0x7fa504072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient(hunting): picked mon.noname-b con 0x7fa504074810 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fa5040b1540 0x7fa5040b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient(hunting): picked mon.noname-a con 0x7fa5040b1540 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa504074180 con 0x7fa5040b4c10 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fa504074c80 con 0x7fa5040b1540 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa504071be0 con 0x7fa504074810 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:18.159 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.154+0000 7fa50935b640 10 monclient(hunting): authenticate will time out at 1282.184913s 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5027fc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa504074810 0x7fa504072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5037fe640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fa5040b1540 0x7fa5040b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5027fc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa504074810 0x7fa504072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:50632/0 (socket says 172.21.15.177:50632) 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5027fc640 1 -- 172.21.15.177:0/698978749 learned_addr learned my addr 172.21.15.177:0/698978749 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5027fc640 10 monclient(hunting): get_auth_request con 0x7fa504074810 auth_method 0 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5027fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.155+0000 7fa5027fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5037fe640 10 monclient(hunting): get_auth_request con 0x7fa5040b1540 auth_method 0 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5037fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.156+0000 7fa5037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 10 monclient(hunting): handle_auth_done global_id 4124 payload 274 2024-07-26T23:34:18.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 1 -- 172.21.15.177:0/698978749 >> v2:172.21.15.139:3300/0 conn(0x7fa5040b1540 msgr2=0x7fa5040b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:18.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 1 --2- 172.21.15.177:0/698978749 >> v2:172.21.15.139:3300/0 conn(0x7fa5040b1540 0x7fa5040b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 1 -- 172.21.15.177:0/698978749 >> v1:172.21.15.139:6789/0 conn(0x7fa5040b4c10 legacy=0x7fa5041416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 10 monclient: _finish_hunting 0 2024-07-26T23:34:18.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 1 monclient: found mon.noname-b 2024-07-26T23:34:18.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:18.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:18.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.158+0000 7fa5027fc640 1 -- 172.21.15.177:0/698978749 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5040725b0 con 0x7fa504074810 2024-07-26T23:34:18.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.160+0000 7fa5027fc640 1 --2- 172.21.15.177:0/698978749 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa504074810 0x7fa504072070 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009870 tx=0x7fa4f802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eda0121f9991b20 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:18.165 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 1 -- 172.21.15.177:0/698978749 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8034070 con 0x7fa504074810 2024-07-26T23:34:18.165 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.165 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:18.165 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: dump: 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: _finish_auth 0 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 1 -- 172.21.15.177:0/698978749 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f8030840 con 0x7fa504074810 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 1 -- 172.21.15.177:0/698978749 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8030c10 con 0x7fa504074810 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa501ffb640 10 monclient: dump: 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.167 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa50935b640 5 monclient: authenticate success, global_id 4124 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa50935b640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:18.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa50935b640 4 set_mon_vals no callback set 2024-07-26T23:34:18.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa50935b640 10 monclient: shutdown 2024-07-26T23:34:18.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa50935b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:18.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa50935b640 1 -- 172.21.15.177:0/698978749 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa504074810 msgr2=0x7fa504072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.164+0000 7fa50935b640 1 --2- 172.21.15.177:0/698978749 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa504074810 0x7fa504072070 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009870 tx=0x7fa4f802b980 comp rx=0 tx=0).stop 2024-07-26T23:34:18.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 -- 172.21.15.177:0/698978749 shutdown_connections 2024-07-26T23:34:18.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 --2- 172.21.15.177:0/698978749 >> v2:172.21.15.139:3300/0 conn(0x7fa5040b1540 0x7fa5040b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 --2- 172.21.15.177:0/698978749 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa504074810 0x7fa504072070 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 -- 172.21.15.177:0/698978749 >> 172.21.15.177:0/698978749 conn(0x7fa5040b7d60 msgr2=0x7fa5040b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 -- 172.21.15.177:0/698978749 shutdown_connections 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 -- 172.21.15.177:0/698978749 wait complete. 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 10 monclient: build_initial_monmap 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 10 monclient: monmap: 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:18.174 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 librados: starting msgr at 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.165+0000 7fa50935b640 1 librados: starting objecter 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 Processor -- start 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 -- start start 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 librados: setting wanted keys 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 librados: calling monclient init 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient: init 2024-07-26T23:34:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa504074810 0x7fa504130ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): picked mon.noname-a con 0x7fa504074810 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5040b1540 0x7fa50413fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5040b1540 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5040b4c10 0x7fa504141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): picked mon.noname-b con 0x7fa5040b4c10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa50414c000 con 0x7fa504074810 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa50413f230 con 0x7fa5040b4c10 2024-07-26T23:34:18.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa50414c180 con 0x7fa5040b1540 2024-07-26T23:34:18.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:18.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.166+0000 7fa50935b640 10 monclient(hunting): authenticate will time out at 1282.196898s 2024-07-26T23:34:18.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5027fc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5040b1540 0x7fa50413fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5037fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5040b4c10 0x7fa504141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa502ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa504074810 0x7fa504130ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5027fc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5040b1540 0x7fa50413fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:50634/0 (socket says 172.21.15.177:50634) 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5027fc640 1 -- 172.21.15.177:0/621853364 learned_addr learned my addr 172.21.15.177:0/621853364 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5027fc640 10 monclient(hunting): get_auth_request con 0x7fa5040b1540 auth_method 0 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5027fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5027fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa502ffd640 10 monclient(hunting): get_auth_request con 0x7fa504074810 auth_method 0 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa502ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa502ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5037fe640 10 monclient(hunting): get_auth_request con 0x7fa5040b4c10 auth_method 0 2024-07-26T23:34:18.178 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5037fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.167+0000 7fa5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa5027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa5027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa5027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa502ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa502ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa502ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa5037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa5037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.168+0000 7fa5037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 10 monclient(hunting): handle_auth_done global_id 4140 payload 824 2024-07-26T23:34:18.179 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 1 -- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5040b1540 msgr2=0x7fa50413fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5040b1540 0x7fa50413fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 1 -- 172.21.15.177:0/621853364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5040b4c10 msgr2=0x7fa504141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5040b4c10 0x7fa504141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 1 monclient: found mon.noname-a 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa502ffd640 1 -- 172.21.15.177:0/621853364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50413c090 con 0x7fa504074810 2024-07-26T23:34:18.180 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.170+0000 7fa5027fc640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5040b1540 0x7fa50413fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T23:34:18.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.173+0000 7fa502ffd640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa504074810 0x7fa504130ec0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400c340 tx=0x7fa4f400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:18.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.173+0000 7fa4eb7fe640 10 client.?.objecter ms_handle_connect 0x7fa504074810 2024-07-26T23:34:18.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.173+0000 7fa4eb7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:18.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 1 -- 172.21.15.177:0/621853364 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4017070 con 0x7fa504074810 2024-07-26T23:34:18.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: dump: 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: _finish_auth 0 2024-07-26T23:34:18.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa50935b640 5 monclient: authenticate success, global_id 4140 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 1 -- 172.21.15.177:0/621853364 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f40052d0 con 0x7fa504074810 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 1 -- 172.21.15.177:0/621853364 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4003810 con 0x7fa504074810 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa50935b640 10 monclient: _renew_subs 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa50935b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50413c830 con 0x7fa504074810 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa4eb7fe640 10 monclient: dump: 2024-07-26T23:34:18.185 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.176+0000 7fa5017fa640 4 set_mon_vals no callback set 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 10 monclient: _renew_subs 2024-07-26T23:34:18.186 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50413cdf0 con 0x7fa504074810 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 1 librados: init done 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.177+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa504072070 con 0x7fa504074810 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.180+0000 7fa4eb7fe640 1 -- 172.21.15.177:0/621853364 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa4f401a070 con 0x7fa504074810 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.180+0000 7fa4eb7fe640 1 -- 172.21.15.177:0/621853364 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fa4f401b3b0 con 0x7fa504074810 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.180+0000 7fa4eb7fe640 10 client.4140.objecter ms_dispatch 0x7fa504000b90 osd_map(6..6 src has 1..6) 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.180+0000 7fa4eb7fe640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.180+0000 7fa4eb7fe640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2024-07-26T23:34:18.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.180+0000 7fa4eb7fe640 20 client.4140.objecter dump_active .. 0 homeless 2024-07-26T23:34:18.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.272+0000 7fa4eb7fe640 1 -- 172.21.15.177:0/621853364 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7fa4f4021020 con 0x7fa504074810 2024-07-26T23:34:18.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.272+0000 7fa4eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.272+0000 7fa4eb7fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:18.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.345+0000 7fa50935b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5a61bbd3-3522-461c-bfd5-ac5d0d6103ca", "id": 4}] 2024-07-26T23:34:18.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.345+0000 7fa50935b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5a61bbd3-3522-461c-bfd5-ac5d0d6103ca", "id": 4}] 2024-07-26T23:34:18.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.345+0000 7fa50935b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:18.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.345+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5a61bbd3-3522-461c-bfd5-ac5d0d6103ca", "id": 4} v 0) -- 0x7fa5040b3b70 con 0x7fa504074810 2024-07-26T23:34:18.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.383+0000 7fa4eb7fe640 1 -- 172.21.15.177:0/621853364 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5a61bbd3-3522-461c-bfd5-ac5d0d6103ca", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fa50413c090 con 0x7fa504074810 2024-07-26T23:34:18.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.383+0000 7fa4eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5a61bbd3-3522-461c-bfd5-ac5d0d6103ca", "id": 4}] 2024-07-26T23:34:18.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.383+0000 7fa4eb7fe640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:18.384 INFO:teuthology.orchestra.run.smithi177.stdout:4 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 10 librados: watch_flush enter 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 10 librados: watch_flush exit 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 20 client.4140.objecter shutdown clearing up homeless session... 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 10 client.4140.objecter successfully canceled tick 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 10 monclient: shutdown 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa504074810 msgr2=0x7fa504130ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa504074810 0x7fa504130ec0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400c340 tx=0x7fa4f400c800 comp rx=0 tx=0).stop 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 shutdown_connections 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5040b1540 0x7fa50413fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5040b4c10 0x7fa504141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 --2- 172.21.15.177:0/621853364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa504074810 0x7fa504130ec0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 >> 172.21.15.177:0/621853364 conn(0x7fa5040b7d60 msgr2=0x7fa5040b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:18.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 shutdown_connections 2024-07-26T23:34:18.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 -- 172.21.15.177:0/621853364 wait complete. 2024-07-26T23:34:18.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.384+0000 7fa50935b640 1 librados: shutdown 2024-07-26T23:34:18.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new add8e38e-b0e0-4fa4-b314-d2e3772398dc 5 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.494+0000 7ff6e0dc0640 10 monclient: get_monmap_and_config 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.494+0000 7ff6e0dc0640 10 monclient: build_initial_monmap 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.494+0000 7ff6e0dc0640 10 monclient: monmap: 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:18.494993+0000 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:18.494993+0000 2024-07-26T23:34:18.495 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:18.496 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.496 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:18.496 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:18.496 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:18.496 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:18.496 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:18.496 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.497 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.494+0000 7ff6e0dc0640 1 Processor -- start 2024-07-26T23:34:18.497 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.494+0000 7ff6e0dc0640 1 -- start start 2024-07-26T23:34:18.497 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.494+0000 7ff6e0dc0640 10 monclient: init 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc1470c0 0x7ff6dc140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient(hunting): picked mon.noname-b con 0x7ff6dc1470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6dc140e90 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6dc0b48d0 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc147ce0 con 0x7ff6dc0b48d0 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc0acaf0 con 0x7ff6dc140e90 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc1477c0 con 0x7ff6dc1470c0 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:18.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.495+0000 7ff6e0dc0640 10 monclient(hunting): authenticate will time out at 1282.525546s 2024-07-26T23:34:18.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.496+0000 7ff6d3fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff6dc140e90 0x7ff6dc0b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:41236/0 (socket says 172.21.15.177:41236) 2024-07-26T23:34:18.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.496+0000 7ff6d3fff640 1 -- 172.21.15.177:0/2092367857 learned_addr learned my addr 172.21.15.177:0/2092367857 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-07-26T23:34:18.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.496+0000 7ff6da59c640 1 --2- 172.21.15.177:0/2092367857 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc1470c0 0x7ff6dc140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.496+0000 7ff6da59c640 10 monclient(hunting): get_auth_request con 0x7ff6dc1470c0 auth_method 0 2024-07-26T23:34:18.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.496+0000 7ff6da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.496+0000 7ff6da59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.496+0000 7ff6da59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.497+0000 7ff6da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.497+0000 7ff6da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.497+0000 7ff6da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 1 -- 172.21.15.177:0/2092367857 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3081367021 0 0) 0x7ff6dc0acaf0 con 0x7ff6dc140e90 2024-07-26T23:34:18.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): _negotiate 2024-07-26T23:34:18.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): my global_id is 4132 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 1 -- 172.21.15.177:0/2092367857 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c8003520 con 0x7ff6dc140e90 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 1 -- 172.21.15.177:0/2092367857 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2358652840 0 0) 0x7ff6dc147ce0 con 0x7ff6dc0b48d0 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): _negotiate 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 10 monclient(hunting): my global_id is 4143 2024-07-26T23:34:18.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6d9d9b640 1 -- 172.21.15.177:0/2092367857 --> v1:172.21.15.139:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6dc0acaf0 con 0x7ff6dc0b48d0 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 10 monclient(hunting): handle_auth_done global_id 4130 payload 274 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 1 -- 172.21.15.177:0/2092367857 >> v1:172.21.15.177:6789/0 conn(0x7ff6dc140e90 legacy=0x7ff6dc0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 1 -- 172.21.15.177:0/2092367857 >> v1:172.21.15.139:6789/0 conn(0x7ff6dc0b48d0 legacy=0x7ff6dc14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 10 monclient: _finish_hunting 0 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 1 monclient: found mon.noname-b 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.498+0000 7ff6da59c640 1 -- 172.21.15.177:0/2092367857 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc1474a0 con 0x7ff6dc1470c0 2024-07-26T23:34:18.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.500+0000 7ff6da59c640 1 --2- 172.21.15.177:0/2092367857 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc1470c0 0x7ff6dc140950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0097d0 tx=0x7ff6cc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbdd6bfd9c4805e9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:18.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 1 -- 172.21.15.177:0/2092367857 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc035020 con 0x7ff6dc1470c0 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: dump: 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.508 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: _finish_auth 0 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 1 -- 172.21.15.177:0/2092367857 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6cc031890 con 0x7ff6dc1470c0 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 1 -- 172.21.15.177:0/2092367857 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc031c60 con 0x7ff6dc1470c0 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6d9d9b640 10 monclient: dump: 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.509 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6e0dc0640 5 monclient: authenticate success, global_id 4130 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6e0dc0640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.505+0000 7ff6e0dc0640 4 set_mon_vals no callback set 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 10 monclient: shutdown 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:18.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2092367857 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc1470c0 msgr2=0x7ff6dc140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 1 --2- 172.21.15.177:0/2092367857 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc1470c0 0x7ff6dc140950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0097d0 tx=0x7ff6cc02b930 comp rx=0 tx=0).stop 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2092367857 shutdown_connections 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 1 --2- 172.21.15.177:0/2092367857 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc1470c0 0x7ff6dc140950 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2092367857 >> 172.21.15.177:0/2092367857 conn(0x7ff6dc0b62c0 msgr2=0x7ff6dc0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2092367857 shutdown_connections 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.506+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2092367857 wait complete. 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 10 monclient: build_initial_monmap 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 10 monclient: monmap: 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:18.511 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:18.512 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.512 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:18.512 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:18.512 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:18.512 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 1 librados: starting msgr at 2024-07-26T23:34:18.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 1 librados: starting objecter 2024-07-26T23:34:18.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 1 Processor -- start 2024-07-26T23:34:18.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 1 -- start start 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 1 librados: setting wanted keys 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 1 librados: calling monclient init 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.507+0000 7ff6e0dc0640 10 monclient: init 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0b48d0 0x7ff6dc143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): picked mon.noname-a con 0x7ff6dc0b48d0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc140e90 0x7ff6dc1dc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): picked mon.noname-c con 0x7ff6dc140e90 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6dc1470c0 0x7ff6dc13f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): picked mon.noname-b con 0x7ff6dc1470c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc141410 con 0x7ff6dc0b48d0 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc074850 con 0x7ff6dc1470c0 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc14f900 con 0x7ff6dc140e90 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6e0dc0640 10 monclient(hunting): authenticate will time out at 1282.538102s 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6d3fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc140e90 0x7ff6dc1dc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6da59c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0b48d0 0x7ff6dc143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6dad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6dc1470c0 0x7ff6dc13f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6d3fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc140e90 0x7ff6dc1dc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:50648/0 (socket says 172.21.15.177:50648) 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6d3fff640 1 -- 172.21.15.177:0/2540393677 learned_addr learned my addr 172.21.15.177:0/2540393677 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:18.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6d3fff640 10 monclient(hunting): get_auth_request con 0x7ff6dc140e90 auth_method 0 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6d3fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.508+0000 7ff6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6dad9d640 10 monclient(hunting): get_auth_request con 0x7ff6dc1470c0 auth_method 0 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6dad9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6da59c640 10 monclient(hunting): get_auth_request con 0x7ff6dc0b48d0 auth_method 0 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6da59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6da59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.509+0000 7ff6da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 1 -- 172.21.15.177:0/2540393677 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc140e90 msgr2=0x7ff6dc1dc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 1 --2- 172.21.15.177:0/2540393677 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc140e90 0x7ff6dc1dc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 1 -- 172.21.15.177:0/2540393677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0b48d0 msgr2=0x7ff6dc143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 1 --2- 172.21.15.177:0/2540393677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0b48d0 0x7ff6dc143630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 10 monclient: _finish_hunting 0 2024-07-26T23:34:18.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 1 monclient: found mon.noname-b 2024-07-26T23:34:18.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:18.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.512+0000 7ff6dad9d640 1 -- 172.21.15.177:0/2540393677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc13f160 con 0x7ff6dc1470c0 2024-07-26T23:34:18.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.514+0000 7ff6dad9d640 1 --2- 172.21.15.177:0/2540393677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6dc1470c0 0x7ff6dc13f7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400c6c0 tx=0x7ff6d400cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:18.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.514+0000 7ff6d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff6dc1470c0 2024-07-26T23:34:18.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.514+0000 7ff6d2ffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:18.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 1 -- 172.21.15.177:0/2540393677 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4003b70 con 0x7ff6dc1470c0 2024-07-26T23:34:18.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: dump: 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.522 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: _finish_auth 0 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 1 -- 172.21.15.177:0/2540393677 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d4003d10 con 0x7ff6dc1470c0 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 1 -- 172.21.15.177:0/2540393677 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4005310 con 0x7ff6dc1470c0 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d2ffd640 10 monclient: dump: 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.523 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 5 monclient: authenticate success, global_id 4135 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 10 monclient: _renew_subs 2024-07-26T23:34:18.524 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6d959a640 4 set_mon_vals no callback set 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc13f3d0 con 0x7ff6dc1470c0 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 10 monclient: _renew_subs 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc13ddf0 con 0x7ff6dc1470c0 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.518+0000 7ff6e0dc0640 1 librados: init done 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.519+0000 7ff6e0dc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.519+0000 7ff6e0dc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.519+0000 7ff6e0dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.519+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6dc140950 con 0x7ff6dc1470c0 2024-07-26T23:34:18.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.521+0000 7ff6d2ffd640 1 -- 172.21.15.177:0/2540393677 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff6d40054b0 con 0x7ff6dc1470c0 2024-07-26T23:34:18.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.521+0000 7ff6d2ffd640 1 -- 172.21.15.177:0/2540393677 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7ff6d40259f0 con 0x7ff6dc1470c0 2024-07-26T23:34:18.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.521+0000 7ff6d2ffd640 10 client.4135.objecter ms_dispatch 0x7ff6dc000b90 osd_map(7..7 src has 1..7) 2024-07-26T23:34:18.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.521+0000 7ff6d2ffd640 3 client.4135.objecter handle_osd_map got epochs [7,7] > 0 2024-07-26T23:34:18.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.521+0000 7ff6d2ffd640 3 client.4135.objecter handle_osd_map decoding full epoch 7 2024-07-26T23:34:18.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.521+0000 7ff6d2ffd640 20 client.4135.objecter dump_active .. 0 homeless 2024-07-26T23:34:18.618 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.617+0000 7ff6d2ffd640 1 -- 172.21.15.177:0/2540393677 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7ff6dc13f3d0 con 0x7ff6dc1470c0 2024-07-26T23:34:18.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.617+0000 7ff6d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.617+0000 7ff6d2ffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:18.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.690+0000 7ff6e0dc0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "add8e38e-b0e0-4fa4-b314-d2e3772398dc", "id": 5}] 2024-07-26T23:34:18.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.690+0000 7ff6e0dc0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "add8e38e-b0e0-4fa4-b314-d2e3772398dc", "id": 5}] 2024-07-26T23:34:18.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.690+0000 7ff6e0dc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.690+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "add8e38e-b0e0-4fa4-b314-d2e3772398dc", "id": 5} v 0) -- 0x7ff6dc13d860 con 0x7ff6dc1470c0 2024-07-26T23:34:18.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.778+0000 7ff6d2ffd640 1 -- 172.21.15.177:0/2540393677 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "add8e38e-b0e0-4fa4-b314-d2e3772398dc", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7ff6d4025c40 con 0x7ff6dc1470c0 2024-07-26T23:34:18.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.778+0000 7ff6d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "add8e38e-b0e0-4fa4-b314-d2e3772398dc", "id": 5}] 2024-07-26T23:34:18.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.778+0000 7ff6d2ffd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:18.780 INFO:teuthology.orchestra.run.smithi177.stdout:5 2024-07-26T23:34:18.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.779+0000 7ff6e0dc0640 10 librados: watch_flush enter 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 10 librados: watch_flush exit 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 20 client.4135.objecter shutdown clearing up homeless session... 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 10 client.4135.objecter successfully canceled tick 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 10 monclient: shutdown 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6dc1470c0 msgr2=0x7ff6dc13f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 --2- 172.21.15.177:0/2540393677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6dc1470c0 0x7ff6dc13f7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400c6c0 tx=0x7ff6d400cb80 comp rx=0 tx=0).stop 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 shutdown_connections 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 --2- 172.21.15.177:0/2540393677 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff6dc140e90 0x7ff6dc1dc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 --2- 172.21.15.177:0/2540393677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6dc1470c0 0x7ff6dc13f7f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 --2- 172.21.15.177:0/2540393677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0b48d0 0x7ff6dc143630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 >> 172.21.15.177:0/2540393677 conn(0x7ff6dc0b62c0 msgr2=0x7ff6dc0b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:18.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 shutdown_connections 2024-07-26T23:34:18.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 -- 172.21.15.177:0/2540393677 wait complete. 2024-07-26T23:34:18.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.780+0000 7ff6e0dc0640 1 librados: shutdown 2024-07-26T23:34:18.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 222ad536-af41-44d8-b9ba-464d0f2c6bb1 6 2024-07-26T23:34:18.891 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.890+0000 7fddccd9d640 10 monclient: get_monmap_and_config 2024-07-26T23:34:18.891 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.890+0000 7fddccd9d640 10 monclient: build_initial_monmap 2024-07-26T23:34:18.891 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.890+0000 7fddccd9d640 10 monclient: monmap: 2024-07-26T23:34:18.891 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:18.891221+0000 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:18.891221+0000 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:18.892 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:18.893 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:18.893 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:18.893 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.890+0000 7fddccd9d640 1 Processor -- start 2024-07-26T23:34:18.901 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.890+0000 7fddccd9d640 1 -- start start 2024-07-26T23:34:18.901 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.890+0000 7fddccd9d640 10 monclient: init 2024-07-26T23:34:18.901 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:18.901 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-07-26T23:34:18.901 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc8140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.901 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fddc81470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:18.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddc8140e90 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:18.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fddc80b48d0 0x7fddc814e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fddc80b48d0 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:18.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddc8147ce0 con 0x7fddc8140e90 2024-07-26T23:34:18.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fddc80acaf0 con 0x7fddc80b48d0 2024-07-26T23:34:18.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fddc81477c0 con 0x7fddc81470c0 2024-07-26T23:34:18.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:18.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.891+0000 7fddccd9d640 10 monclient(hunting): authenticate will time out at 1282.921793s 2024-07-26T23:34:18.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc6d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fddc80b48d0 0x7fddc814e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc659c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc8140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc659c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc8140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:57494/0 (socket says 172.21.15.177:57494) 2024-07-26T23:34:18.904 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc659c640 1 -- 172.21.15.177:0/3150677144 learned_addr learned my addr 172.21.15.177:0/3150677144 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:18.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc659c640 10 monclient(hunting): get_auth_request con 0x7fddc81470c0 auth_method 0 2024-07-26T23:34:18.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.892+0000 7fddc659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc6d9d640 10 monclient(hunting): get_auth_request con 0x7fddc80b48d0 auth_method 0 2024-07-26T23:34:18.905 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.907 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.908 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.908 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.908 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.893+0000 7fddc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.908 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2024-07-26T23:34:18.908 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 1 -- 172.21.15.177:0/3150677144 >> v2:172.21.15.177:3300/0 conn(0x7fddc80b48d0 msgr2=0x7fddc814e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:18.909 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 1 --2- 172.21.15.177:0/3150677144 >> v2:172.21.15.177:3300/0 conn(0x7fddc80b48d0 0x7fddc814e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.909 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 1 -- 172.21.15.177:0/3150677144 >> v1:172.21.15.177:6789/0 conn(0x7fddc8140e90 legacy=0x7fddc80b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.909 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 10 monclient: _finish_hunting 0 2024-07-26T23:34:18.909 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 1 monclient: found mon.noname-b 2024-07-26T23:34:18.910 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:18.910 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:18.910 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.894+0000 7fddc659c640 1 -- 172.21.15.177:0/3150677144 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc81474a0 con 0x7fddc81470c0 2024-07-26T23:34:18.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.896+0000 7fddc659c640 1 --2- 172.21.15.177:0/3150677144 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc8140950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009870 tx=0x7fddb802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c49ad1b011a37e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 1 -- 172.21.15.177:0/3150677144 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8034070 con 0x7fddc81470c0 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: dump: 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.922 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: _finish_auth 0 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: _check_auth_tickets 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 1 -- 172.21.15.177:0/3150677144 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb8030840 con 0x7fddc81470c0 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 1 -- 172.21.15.177:0/3150677144 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8030c10 con 0x7fddc81470c0 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:18.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddc559a640 10 monclient: dump: 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddccd9d640 5 monclient: authenticate success, global_id 4136 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddccd9d640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddccd9d640 4 set_mon_vals no callback set 2024-07-26T23:34:18.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.901+0000 7fddccd9d640 10 monclient: shutdown 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 -- 172.21.15.177:0/3150677144 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 msgr2=0x7fddc8140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 --2- 172.21.15.177:0/3150677144 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc8140950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009870 tx=0x7fddb802b980 comp rx=0 tx=0).stop 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 -- 172.21.15.177:0/3150677144 shutdown_connections 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 --2- 172.21.15.177:0/3150677144 >> v2:172.21.15.177:3300/0 conn(0x7fddc80b48d0 0x7fddc814e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 --2- 172.21.15.177:0/3150677144 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc8140950 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 -- 172.21.15.177:0/3150677144 >> 172.21.15.177:0/3150677144 conn(0x7fddc80b62c0 msgr2=0x7fddc80b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 -- 172.21.15.177:0/3150677144 shutdown_connections 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 1 -- 172.21.15.177:0/3150677144 wait complete. 2024-07-26T23:34:18.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.902+0000 7fddccd9d640 10 monclient: build_initial_monmap 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient: monmap: 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:18.926 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 librados: starting msgr at 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 librados: starting objecter 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 Processor -- start 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 -- start start 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 librados: setting wanted keys 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 librados: calling monclient init 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient: init 2024-07-26T23:34:18.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc80b48d0 0x7fddc813b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fddc80b48d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddc8140e90 0x7fddc81438b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fddc8140e90 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc81d8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fddc81470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fddc8141410 con 0x7fddc8140e90 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fddc8074850 con 0x7fddc80b48d0 2024-07-26T23:34:18.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fddc814e720 con 0x7fddc81470c0 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.903+0000 7fddccd9d640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddccd9d640 10 monclient(hunting): authenticate will time out at 1282.934006s 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc80b48d0 0x7fddc813b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc5d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddc8140e90 0x7fddc81438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc6d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc81d8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc80b48d0 0x7fddc813b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:54532/0 (socket says 172.21.15.177:54532) 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc659c640 1 -- 172.21.15.177:0/2080039522 learned_addr learned my addr 172.21.15.177:0/2080039522 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc659c640 10 monclient(hunting): get_auth_request con 0x7fddc80b48d0 auth_method 0 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.904+0000 7fddc659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc6d9d640 10 monclient(hunting): get_auth_request con 0x7fddc81470c0 auth_method 0 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc6d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc5d9b640 10 monclient(hunting): get_auth_request con 0x7fddc8140e90 auth_method 0 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc5d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.930 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.931 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.931 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.931 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.931 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:18.931 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:18.931 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.905+0000 7fddc5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 1 -- 172.21.15.177:0/2080039522 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 msgr2=0x7fddc81d8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 1 --2- 172.21.15.177:0/2080039522 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc81d8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 1 -- 172.21.15.177:0/2080039522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddc8140e90 msgr2=0x7fddc81438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 1 --2- 172.21.15.177:0/2080039522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddc8140e90 0x7fddc81438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 10 monclient: _finish_hunting 0 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 1 monclient: found mon.noname-b 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.908+0000 7fddc659c640 1 -- 172.21.15.177:0/2080039522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc8142d10 con 0x7fddc80b48d0 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.910+0000 7fddc659c640 1 --2- 172.21.15.177:0/2080039522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc80b48d0 0x7fddc813b510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddb802b9c0 tx=0x7fddb80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.910+0000 7fddb6ffd640 10 client.?.objecter ms_handle_connect 0x7fddc80b48d0 2024-07-26T23:34:18.932 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.910+0000 7fddb6ffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 1 -- 172.21.15.177:0/2080039522 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8034040 con 0x7fddc80b48d0 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: dump: 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.933 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: _finish_auth 0 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:18.934 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 1 -- 172.21.15.177:0/2080039522 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb8030640 con 0x7fddc80b48d0 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 1 -- 172.21.15.177:0/2080039522 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb80423f0 con 0x7fddc80b48d0 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddb6ffd640 10 monclient: dump: 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:18.935 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:18.936 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 5 monclient: authenticate success, global_id 4141 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 10 monclient: _renew_subs 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddc4d99640 4 set_mon_vals no callback set 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc81434f0 con 0x7fddc80b48d0 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 10 monclient: _renew_subs 2024-07-26T23:34:18.937 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.938 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc81d92d0 con 0x7fddc80b48d0 2024-07-26T23:34:18.938 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.913+0000 7fddccd9d640 1 librados: init done 2024-07-26T23:34:18.938 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.914+0000 7fddccd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.938 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.914+0000 7fddccd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:18.938 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.914+0000 7fddccd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:18.938 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.914+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc8140950 con 0x7fddc80b48d0 2024-07-26T23:34:18.938 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.917+0000 7fddb6ffd640 1 -- 172.21.15.177:0/2080039522 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fddb8037070 con 0x7fddc80b48d0 2024-07-26T23:34:18.939 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.917+0000 7fddb6ffd640 1 -- 172.21.15.177:0/2080039522 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fddb8042c60 con 0x7fddc80b48d0 2024-07-26T23:34:18.939 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.917+0000 7fddb6ffd640 10 client.4141.objecter ms_dispatch 0x7fddc8000b90 osd_map(8..8 src has 1..8) 2024-07-26T23:34:18.939 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.917+0000 7fddb6ffd640 3 client.4141.objecter handle_osd_map got epochs [8,8] > 0 2024-07-26T23:34:18.939 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.917+0000 7fddb6ffd640 3 client.4141.objecter handle_osd_map decoding full epoch 8 2024-07-26T23:34:18.939 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:18.917+0000 7fddb6ffd640 20 client.4141.objecter dump_active .. 0 homeless 2024-07-26T23:34:19.012 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.011+0000 7fddb6ffd640 1 -- 172.21.15.177:0/2080039522 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7fddb80307f0 con 0x7fddc80b48d0 2024-07-26T23:34:19.013 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.011+0000 7fddb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:19.013 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.011+0000 7fddb6ffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:19.086 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.085+0000 7fddccd9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "222ad536-af41-44d8-b9ba-464d0f2c6bb1", "id": 6}] 2024-07-26T23:34:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.085+0000 7fddccd9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "222ad536-af41-44d8-b9ba-464d0f2c6bb1", "id": 6}] 2024-07-26T23:34:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.085+0000 7fddccd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:19.087 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.085+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "222ad536-af41-44d8-b9ba-464d0f2c6bb1", "id": 6} v 0) -- 0x7fddc806b560 con 0x7fddc80b48d0 2024-07-26T23:34:19.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.180+0000 7fddb6ffd640 1 -- 172.21.15.177:0/2080039522 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "222ad536-af41-44d8-b9ba-464d0f2c6bb1", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fddb8042590 con 0x7fddc80b48d0 2024-07-26T23:34:19.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.180+0000 7fddb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "222ad536-af41-44d8-b9ba-464d0f2c6bb1", "id": 6}] 2024-07-26T23:34:19.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.180+0000 7fddb6ffd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:19.182 INFO:teuthology.orchestra.run.smithi177.stdout:6 2024-07-26T23:34:19.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 10 librados: watch_flush enter 2024-07-26T23:34:19.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 10 librados: watch_flush exit 2024-07-26T23:34:19.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-07-26T23:34:19.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 10 client.4141.objecter successfully canceled tick 2024-07-26T23:34:19.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 10 monclient: shutdown 2024-07-26T23:34:19.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc80b48d0 msgr2=0x7fddc813b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 --2- 172.21.15.177:0/2080039522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc80b48d0 0x7fddc813b510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fddb802b9c0 tx=0x7fddb80046c0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 shutdown_connections 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 --2- 172.21.15.177:0/2080039522 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fddc81470c0 0x7fddc81d8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 --2- 172.21.15.177:0/2080039522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fddc8140e90 0x7fddc81438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 --2- 172.21.15.177:0/2080039522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc80b48d0 0x7fddc813b510 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 >> 172.21.15.177:0/2080039522 conn(0x7fddc80b62c0 msgr2=0x7fddc8069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 shutdown_connections 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 -- 172.21.15.177:0/2080039522 wait complete. 2024-07-26T23:34:19.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.181+0000 7fddccd9d640 1 librados: shutdown 2024-07-26T23:34:19.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 5cabe1b4-6f95-4145-8a6f-c92e149c2e83 7 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.290+0000 7f12e6257640 10 monclient: get_monmap_and_config 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.290+0000 7f12e6257640 10 monclient: build_initial_monmap 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.290+0000 7f12e6257640 10 monclient: monmap: 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:34:19.291699+0000 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:34:19.291699+0000 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:19.292 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:19.293 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:19.293 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:19.293 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:19.293 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:19.293 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:19.293 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:19.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.291+0000 7f12e6257640 1 Processor -- start 2024-07-26T23:34:19.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.291+0000 7f12e6257640 1 -- start start 2024-07-26T23:34:19.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.291+0000 7f12e6257640 10 monclient: init 2024-07-26T23:34:19.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:19.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-07-26T23:34:19.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f12e01470c0 0x7f12e0140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient(hunting): picked mon.noname-a con 0x7f12e01470c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient(hunting): picked mon.noname-b con 0x7f12e0140e90 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12e00b4700 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e0147ce0 con 0x7f12e00b4700 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f12e00acaf0 con 0x7f12e01470c0 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f12e01477c0 con 0x7f12e0140e90 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.292+0000 7f12e6257640 10 monclient(hunting): authenticate will time out at 1283.322505s 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12dffff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f12e00b4700 0x7f12e01438d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:36662/0 (socket says 172.21.15.177:36662) 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.296 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12dffff640 1 -- 172.21.15.177:0/1665414405 learned_addr learned my addr 172.21.15.177:0/1665414405 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 10 monclient(hunting): get_auth_request con 0x7f12e0140e90 auth_method 0 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12df7fe640 1 --2- 172.21.15.177:0/1665414405 >> v2:172.21.15.139:3300/0 conn(0x7f12e01470c0 0x7f12e0140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12df7fe640 10 monclient(hunting): get_auth_request con 0x7f12e01470c0 auth_method 0 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12df7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.293+0000 7f12d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.297 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.294+0000 7f12df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.298 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.294+0000 7f12df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.298 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.294+0000 7f12df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 1 -- 172.21.15.177:0/1665414405 >> v2:172.21.15.139:3300/0 conn(0x7f12e01470c0 msgr2=0x7f12e0140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 1 --2- 172.21.15.177:0/1665414405 >> v2:172.21.15.139:3300/0 conn(0x7f12e01470c0 0x7f12e0140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 1 -- 172.21.15.177:0/1665414405 >> v1:172.21.15.177:6789/0 conn(0x7f12e00b4700 legacy=0x7f12e01438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 1 monclient: found mon.noname-b 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.295+0000 7f12d7fff640 1 -- 172.21.15.177:0/1665414405 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e00b4b90 con 0x7f12e0140e90 2024-07-26T23:34:19.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.297+0000 7f12d7fff640 1 --2- 172.21.15.177:0/1665414405 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00b41c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f12c80097a0 tx=0x7f12c802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=984fc873daa5b54f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12deffd640 1 -- 172.21.15.177:0/1665414405 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c80315a0 con 0x7f12e0140e90 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12deffd640 10 monclient: dump: 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.301 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12deffd640 10 monclient: _finish_auth 0 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12deffd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12e6257640 5 monclient: authenticate success, global_id 4142 2024-07-26T23:34:19.302 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.300+0000 7f12e6257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12deffd640 1 -- 172.21.15.177:0/1665414405 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c8037070 con 0x7f12e0140e90 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12deffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12deffd640 1 -- 172.21.15.177:0/1665414405 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c8031960 con 0x7f12e0140e90 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 4 set_mon_vals no callback set 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:19.303 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12deffd640 10 monclient: dump: 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.304 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 10 monclient: shutdown 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 -- 172.21.15.177:0/1665414405 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 msgr2=0x7f12e00b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 --2- 172.21.15.177:0/1665414405 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00b41c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f12c80097a0 tx=0x7f12c802b840 comp rx=0 tx=0).stop 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 -- 172.21.15.177:0/1665414405 shutdown_connections 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 --2- 172.21.15.177:0/1665414405 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00b41c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 --2- 172.21.15.177:0/1665414405 >> v2:172.21.15.139:3300/0 conn(0x7f12e01470c0 0x7f12e0140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 -- 172.21.15.177:0/1665414405 >> 172.21.15.177:0/1665414405 conn(0x7f12e00b62c0 msgr2=0x7f12e00b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 -- 172.21.15.177:0/1665414405 shutdown_connections 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.302+0000 7f12e6257640 1 -- 172.21.15.177:0/1665414405 wait complete. 2024-07-26T23:34:19.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.303+0000 7f12e6257640 10 monclient: build_initial_monmap 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.303+0000 7f12e6257640 10 monclient: monmap: 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.303+0000 7f12e6257640 1 librados: starting msgr at 2024-07-26T23:34:19.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.303+0000 7f12e6257640 1 librados: starting objecter 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 Processor -- start 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 -- start start 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 librados: setting wanted keys 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 librados: calling monclient init 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient: init 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e00b4700 0x7f12e01d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): picked mon.noname-b con 0x7f12e00b4700 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): picked mon.noname-c con 0x7f12e0140e90 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:19.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01470c0 0x7f12e00af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): picked mon.noname-a con 0x7f12e01470c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f12e0074850 con 0x7f12e01470c0 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12e00699f0 con 0x7f12e00b4700 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f12e0141410 con 0x7f12e0140e90 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.304+0000 7f12e6257640 10 monclient(hunting): authenticate will time out at 1283.334669s 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12d7fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12df7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e00b4700 0x7f12e01d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12dffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01470c0 0x7f12e00af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12d7fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00ae970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.177:57526/0 (socket says 172.21.15.177:57526) 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12d7fff640 1 -- 172.21.15.177:0/2695962513 learned_addr learned my addr 172.21.15.177:0/2695962513 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12d7fff640 10 monclient(hunting): get_auth_request con 0x7f12e0140e90 auth_method 0 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12d7fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12df7fe640 10 monclient(hunting): get_auth_request con 0x7f12e00b4700 auth_method 0 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12df7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12dffff640 10 monclient(hunting): get_auth_request con 0x7f12e01470c0 auth_method 0 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12dffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.305+0000 7f12dffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.306+0000 7f12d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.315 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2024-07-26T23:34:19.315 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 1 -- 172.21.15.177:0/2695962513 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 msgr2=0x7f12e00ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.315 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 1 --2- 172.21.15.177:0/2695962513 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 1 -- 172.21.15.177:0/2695962513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e00b4700 msgr2=0x7f12e01d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 1 --2- 172.21.15.177:0/2695962513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e00b4700 0x7f12e01d02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 1 monclient: found mon.noname-a 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.308+0000 7f12dffff640 1 -- 172.21.15.177:0/2695962513 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e00aeeb0 con 0x7f12e01470c0 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.311+0000 7f12dffff640 1 --2- 172.21.15.177:0/2695962513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01470c0 0x7f12e00af6f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12d800cae0 tx=0x7f12d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.311+0000 7f12dcff9640 10 client.?.objecter ms_handle_connect 0x7f12e01470c0 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.311+0000 7f12dcff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12dcff9640 1 -- 172.21.15.177:0/2695962513 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8017070 con 0x7f12e01470c0 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12dcff9640 10 monclient: dump: 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12dcff9640 10 monclient: _finish_auth 0 2024-07-26T23:34:19.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12dcff9640 10 monclient: _check_auth_tickets 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 5 monclient: authenticate success, global_id 4155 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 10 monclient: _renew_subs 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e00b1040 con 0x7f12e01470c0 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 10 monclient: _renew_subs 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e01dc5a0 con 0x7f12e01470c0 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.313+0000 7f12e6257640 1 librados: init done 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12e6257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12e6257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:19.318 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12e6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e00b41c0 con 0x7f12e01470c0 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12dcff9640 1 -- 172.21.15.177:0/2695962513 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d800abe0 con 0x7f12e01470c0 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12dcff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12dcff9640 1 -- 172.21.15.177:0/2695962513 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8021060 con 0x7f12e01470c0 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12dcff9640 10 monclient: dump: 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.319 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-07-26T23:34:19.320 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.320 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.320 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.320 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-26T23:34:19.320 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.314+0000 7f12de7fc640 4 set_mon_vals no callback set 2024-07-26T23:34:19.411 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.410+0000 7f12dcff9640 1 -- 172.21.15.177:0/2695962513 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f12d801a070 con 0x7f12e01470c0 2024-07-26T23:34:19.411 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.410+0000 7f12dcff9640 1 -- 172.21.15.177:0/2695962513 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f12d8012c10 con 0x7f12e01470c0 2024-07-26T23:34:19.411 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.410+0000 7f12dcff9640 10 client.4155.objecter ms_dispatch 0x7f12e0000b90 osd_map(9..9 src has 1..9) 2024-07-26T23:34:19.411 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.410+0000 7f12dcff9640 3 client.4155.objecter handle_osd_map got epochs [9,9] > 0 2024-07-26T23:34:19.411 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.410+0000 7f12dcff9640 3 client.4155.objecter handle_osd_map decoding full epoch 9 2024-07-26T23:34:19.412 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.410+0000 7f12dcff9640 20 client.4155.objecter dump_active .. 0 homeless 2024-07-26T23:34:19.415 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.414+0000 7f12dcff9640 1 -- 172.21.15.177:0/2695962513 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7f12d8012540 con 0x7f12e01470c0 2024-07-26T23:34:19.415 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.414+0000 7f12dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:19.415 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.414+0000 7f12dcff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:19.489 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.487+0000 7f12e6257640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5cabe1b4-6f95-4145-8a6f-c92e149c2e83", "id": 7}] 2024-07-26T23:34:19.489 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.487+0000 7f12e6257640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5cabe1b4-6f95-4145-8a6f-c92e149c2e83", "id": 7}] 2024-07-26T23:34:19.489 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.487+0000 7f12e6257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:19.489 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.487+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5cabe1b4-6f95-4145-8a6f-c92e149c2e83", "id": 7} v 0) -- 0x7f12e01403e0 con 0x7f12e01470c0 2024-07-26T23:34:19.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.615+0000 7f12dcff9640 1 -- 172.21.15.177:0/2695962513 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5cabe1b4-6f95-4145-8a6f-c92e149c2e83", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f12d8012810 con 0x7f12e01470c0 2024-07-26T23:34:19.617 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.615+0000 7f12dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5cabe1b4-6f95-4145-8a6f-c92e149c2e83", "id": 7}] 2024-07-26T23:34:19.617 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.615+0000 7f12dcff9640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:19.617 INFO:teuthology.orchestra.run.smithi177.stdout:7 2024-07-26T23:34:19.618 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 10 librados: watch_flush enter 2024-07-26T23:34:19.618 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 10 librados: watch_flush exit 2024-07-26T23:34:19.618 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 20 client.4155.objecter shutdown clearing up homeless session... 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 10 client.4155.objecter successfully canceled tick 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 10 monclient: shutdown 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01470c0 msgr2=0x7f12e00af6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 --2- 172.21.15.177:0/2695962513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01470c0 0x7f12e00af6f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12d800cae0 tx=0x7f12d800a060 comp rx=0 tx=0).stop 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 shutdown_connections 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 --2- 172.21.15.177:0/2695962513 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f12e0140e90 0x7f12e00ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 --2- 172.21.15.177:0/2695962513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12e01470c0 0x7f12e00af6f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 --2- 172.21.15.177:0/2695962513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e00b4700 0x7f12e01d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 >> 172.21.15.177:0/2695962513 conn(0x7f12e00b62c0 msgr2=0x7f12e00b84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 shutdown_connections 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 -- 172.21.15.177:0/2695962513 wait complete. 2024-07-26T23:34:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-07-26T23:34:19.616+0000 7f12e6257640 1 librados: shutdown 2024-07-26T23:34:19.628 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.628 INFO:tasks.ceph.osd.0:Restarting daemon 2024-07-26T23:34:19.628 DEBUG:teuthology.orchestra.run.smithi139:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-07-26T23:34:19.630 INFO:tasks.ceph.osd.0:Started 2024-07-26T23:34:19.631 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.631 INFO:tasks.ceph.osd.1:Restarting daemon 2024-07-26T23:34:19.631 DEBUG:teuthology.orchestra.run.smithi139:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-07-26T23:34:19.633 INFO:tasks.ceph.osd.1:Started 2024-07-26T23:34:19.633 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.633 INFO:tasks.ceph.osd.2:Restarting daemon 2024-07-26T23:34:19.633 DEBUG:teuthology.orchestra.run.smithi139:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-07-26T23:34:19.635 INFO:tasks.ceph.osd.2:Started 2024-07-26T23:34:19.635 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.635 INFO:tasks.ceph.osd.3:Restarting daemon 2024-07-26T23:34:19.636 DEBUG:teuthology.orchestra.run.smithi139:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-07-26T23:34:19.638 INFO:tasks.ceph.osd.3:Started 2024-07-26T23:34:19.638 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.638 INFO:tasks.ceph.osd.4:Restarting daemon 2024-07-26T23:34:19.638 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-07-26T23:34:19.640 INFO:tasks.ceph.osd.4:Started 2024-07-26T23:34:19.640 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.640 INFO:tasks.ceph.osd.5:Restarting daemon 2024-07-26T23:34:19.640 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-07-26T23:34:19.642 INFO:tasks.ceph.osd.5:Started 2024-07-26T23:34:19.643 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.643 INFO:tasks.ceph.osd.6:Restarting daemon 2024-07-26T23:34:19.643 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-07-26T23:34:19.645 INFO:tasks.ceph.osd.6:Started 2024-07-26T23:34:19.645 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-07-26T23:34:19.645 INFO:tasks.ceph.osd.7:Restarting daemon 2024-07-26T23:34:19.645 DEBUG:teuthology.orchestra.run.smithi177:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-07-26T23:34:19.647 INFO:tasks.ceph.osd.7:Started 2024-07-26T23:34:19.648 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T23:34:19.792 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.790+0000 7ff94cf4d640 10 monclient: get_monmap_and_config 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.790+0000 7ff94cf4d640 10 monclient: build_initial_monmap 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.790+0000 7ff94cf4d640 10 monclient: monmap: 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:34:19.791161+0000 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:34:19.791161+0000 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:19.793 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:19.794 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:19.794 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:19.794 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:19.794 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:19.794 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:19.794 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:19.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.792+0000 7ff94cf4d640 1 Processor -- start 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.792+0000 7ff94cf4d640 1 -- start start 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.792+0000 7ff94cf4d640 10 monclient: init 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff948142df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9480b1240 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff9480aeec0 0x7ff9480acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9480aeec0 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9480acfe0 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:19.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9480ae550 con 0x7ff9480acfe0 2024-07-26T23:34:19.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff9480b1620 con 0x7ff9480aeec0 2024-07-26T23:34:19.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff9480ae7c0 con 0x7ff9480b1240 2024-07-26T23:34:19.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:19.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.793+0000 7ff94cf4d640 10 monclient(hunting): authenticate will time out at 1292.829562s 2024-07-26T23:34:19.841 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.839+0000 7ff946d9d640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7ff9480acfe0 0x7ff9480741a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:49478/0 (socket says 172.21.15.139:49478) 2024-07-26T23:34:19.841 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.839+0000 7ff946d9d640 1 -- 172.21.15.139:0/986241030 learned_addr learned my addr 172.21.15.139:0/986241030 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:34:19.841 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.840+0000 7ff94659c640 1 --2- 172.21.15.139:0/986241030 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff948142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.889 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.888+0000 7ff94659c640 10 monclient(hunting): get_auth_request con 0x7ff9480b1240 auth_method 0 2024-07-26T23:34:19.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.888+0000 7ff94659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.888+0000 7ff94659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.888+0000 7ff94659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.890+0000 7ff94659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.890+0000 7ff94659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.890+0000 7ff94659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 10 monclient(hunting): handle_auth_done global_id 4148 payload 274 2024-07-26T23:34:19.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 1 -- 172.21.15.139:0/986241030 >> v2:172.21.15.177:3300/0 conn(0x7ff9480aeec0 msgr2=0x7ff9480acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:19.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 1 --2- 172.21.15.139:0/986241030 >> v2:172.21.15.177:3300/0 conn(0x7ff9480aeec0 0x7ff9480acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 1 -- 172.21.15.139:0/986241030 >> v1:172.21.15.139:6789/0 conn(0x7ff9480acfe0 legacy=0x7ff9480741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 10 monclient: _finish_hunting 0 2024-07-26T23:34:19.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 1 monclient: found mon.noname-b 2024-07-26T23:34:19.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:19.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:19.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.892+0000 7ff94659c640 1 -- 172.21.15.139:0/986241030 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9480aeaa0 con 0x7ff9480b1240 2024-07-26T23:34:19.898 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.897+0000 7ff94659c640 1 --2- 172.21.15.139:0/986241030 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff948142df0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a940 tx=0x7ff94002ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad32c5d68becb25a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:19.906 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 1 -- 172.21.15.139:0/986241030 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94003d070 con 0x7ff9480b1240 2024-07-26T23:34:19.906 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.906 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:19.906 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: dump: 2024-07-26T23:34:19.906 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:19.906 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.906 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: _finish_auth 0 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: _check_auth_tickets 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 1 -- 172.21.15.139:0/986241030 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff940030d30 con 0x7ff9480b1240 2024-07-26T23:34:19.907 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94cf4d640 5 monclient: authenticate success, global_id 4148 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94cf4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94cf4d640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94cf4d640 4 set_mon_vals no callback set 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 1 -- 172.21.15.139:0/986241030 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94000b040 con 0x7ff9480b1240 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.903+0000 7ff94559a640 10 monclient: dump: 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.908 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 10 monclient: shutdown 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 1 -- 172.21.15.139:0/986241030 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 msgr2=0x7ff948142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 1 --2- 172.21.15.139:0/986241030 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff948142df0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a940 tx=0x7ff94002ff80 comp rx=0 tx=0).stop 2024-07-26T23:34:19.909 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 1 -- 172.21.15.139:0/986241030 shutdown_connections 2024-07-26T23:34:19.910 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 1 --2- 172.21.15.139:0/986241030 >> v2:172.21.15.177:3300/0 conn(0x7ff9480aeec0 0x7ff9480acaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.910 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 1 --2- 172.21.15.139:0/986241030 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff948142df0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.910 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.904+0000 7ff94cf4d640 1 -- 172.21.15.139:0/986241030 >> 172.21.15.139:0/986241030 conn(0x7ff9480b1c10 msgr2=0x7ff9480b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:19.910 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.905+0000 7ff94cf4d640 1 -- 172.21.15.139:0/986241030 shutdown_connections 2024-07-26T23:34:19.911 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.905+0000 7ff94cf4d640 1 -- 172.21.15.139:0/986241030 wait complete. 2024-07-26T23:34:19.911 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 10 monclient: build_initial_monmap 2024-07-26T23:34:19.911 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 10 monclient: monmap: 2024-07-26T23:34:19.911 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 1 librados: starting msgr at 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 1 librados: starting objecter 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 1 Processor -- start 2024-07-26T23:34:19.912 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 1 -- start start 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 1 librados: setting wanted keys 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 1 librados: calling monclient init 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.906+0000 7ff94cf4d640 10 monclient: init 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9480aeec0 0x7ff94813ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9480aeec0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff9481d8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9480b1240 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff948140500 0x7ff9481d8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): picked mon.noname-a con 0x7ff948140500 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff9480b28b0 con 0x7ff948140500 2024-07-26T23:34:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9480b2730 con 0x7ff9480aeec0 2024-07-26T23:34:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff9480748b0 con 0x7ff9480b1240 2024-07-26T23:34:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.907+0000 7ff94cf4d640 10 monclient(hunting): authenticate will time out at 1292.943938s 2024-07-26T23:34:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.910+0000 7ff945d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff9481d8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.915 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.911+0000 7ff945d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff9481d8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:51328/0 (socket says 172.21.15.139:51328) 2024-07-26T23:34:19.915 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.911+0000 7ff945d9b640 1 -- 172.21.15.139:0/981582182 learned_addr learned my addr 172.21.15.139:0/981582182 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:19.915 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.911+0000 7ff945d9b640 10 monclient(hunting): get_auth_request con 0x7ff9480b1240 auth_method 0 2024-07-26T23:34:19.915 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.911+0000 7ff945d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.915 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.911+0000 7ff945d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.915 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.911+0000 7ff945d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.912+0000 7ff94659c640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9480aeec0 0x7ff94813ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.913+0000 7ff945d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.913+0000 7ff945d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.913+0000 7ff945d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.914+0000 7ff94659c640 10 monclient(hunting): get_auth_request con 0x7ff9480aeec0 auth_method 0 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.914+0000 7ff94659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.914+0000 7ff94659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:19.916 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.914+0000 7ff94659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:19.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.917+0000 7ff94659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:19.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.917+0000 7ff94659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:19.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.917+0000 7ff94659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:19.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 10 monclient(hunting): handle_auth_done global_id 4151 payload 824 2024-07-26T23:34:19.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 1 -- 172.21.15.139:0/981582182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9480aeec0 msgr2=0x7ff94813ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9480aeec0 0x7ff94813ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 1 -- 172.21.15.139:0/981582182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff948140500 msgr2=0x7ff9481d8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:19.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff948140500 0x7ff9481d8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:19.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 10 monclient: _finish_hunting 0 2024-07-26T23:34:19.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 1 monclient: found mon.noname-c 2024-07-26T23:34:19.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:19.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:34:19.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.918+0000 7ff945d9b640 1 -- 172.21.15.139:0/981582182 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94813f5c0 con 0x7ff9480b1240 2024-07-26T23:34:19.926 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.924+0000 7ff945d9b640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff9481d8640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff93800bab0 tx=0x7ff93800bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:19.926 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.924+0000 7ff936ffd640 10 client.?.objecter ms_handle_connect 0x7ff9480b1240 2024-07-26T23:34:19.926 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.924+0000 7ff936ffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 1 -- 172.21.15.139:0/981582182 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938019070 con 0x7ff9480b1240 2024-07-26T23:34:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 10 monclient: dump: 2024-07-26T23:34:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 10 monclient: _finish_auth 0 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff936ffd640 1 -- 172.21.15.139:0/981582182 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff93800cc20 con 0x7ff9480b1240 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.933+0000 7ff94cf4d640 5 monclient: authenticate success, global_id 4151 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.934+0000 7ff94cf4d640 10 monclient: _renew_subs 2024-07-26T23:34:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.934+0000 7ff94cf4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.934+0000 7ff94cf4d640 1 -- 172.21.15.139:0/981582182 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94813f830 con 0x7ff9480b1240 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.934+0000 7ff936ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.934+0000 7ff944d99640 4 set_mon_vals no callback set 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.934+0000 7ff94cf4d640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff94cf4d640 10 monclient: _renew_subs 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff94cf4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff94cf4d640 1 -- 172.21.15.139:0/981582182 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9481e90a0 con 0x7ff9480b1240 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff94cf4d640 1 librados: init done 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff936ffd640 1 -- 172.21.15.139:0/981582182 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938014440 con 0x7ff9480b1240 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff936ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff936ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.935+0000 7ff936ffd640 10 monclient: dump: 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:34:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:34:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.937+0000 7ff94cf4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:19.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.937+0000 7ff94cf4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:19.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.937+0000 7ff94cf4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:19.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.937+0000 7ff94cf4d640 1 -- 172.21.15.139:0/981582182 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9480acaa0 con 0x7ff9480b1240 2024-07-26T23:34:19.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:19.961+0000 7ff936ffd640 1 -- 172.21.15.139:0/981582182 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 2) ==== 49591+0+0 (secure 0 0 0) 0x7ff93801c070 con 0x7ff9480b1240 2024-07-26T23:34:20.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.002+0000 7ff936ffd640 1 -- 172.21.15.139:0/981582182 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ff93801dc60 con 0x7ff9480b1240 2024-07-26T23:34:20.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.002+0000 7ff936ffd640 10 client.4151.objecter ms_dispatch 0x7ff948000b90 osd_map(10..10 src has 1..10) 2024-07-26T23:34:20.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.002+0000 7ff936ffd640 3 client.4151.objecter handle_osd_map got epochs [10,10] > 0 2024-07-26T23:34:20.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.002+0000 7ff936ffd640 3 client.4151.objecter handle_osd_map decoding full epoch 10 2024-07-26T23:34:20.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.002+0000 7ff936ffd640 20 client.4151.objecter dump_active .. 0 homeless 2024-07-26T23:34:20.150 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.144+0000 7ff936ffd640 1 -- 172.21.15.139:0/981582182 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7ff938023020 con 0x7ff9480b1240 2024-07-26T23:34:20.150 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.144+0000 7ff936ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:20.150 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.144+0000 7ff936ffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:20.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.271+0000 7ff94cf4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:20.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.271+0000 7ff94cf4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:20.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.271+0000 7ff94cf4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:20.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.271+0000 7ff94cf4d640 1 -- 172.21.15.139:0/981582182 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff948073d50 con 0x7ff9480b1240 2024-07-26T23:34:20.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.362+0000 7ff936ffd640 1 -- 172.21.15.139:0/981582182 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ff93801d740 con 0x7ff9480b1240 2024-07-26T23:34:20.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.362+0000 7ff936ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:20.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.362+0000 7ff936ffd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:34:20.365 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":10,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:34:19.494450+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:34:20.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 10 librados: watch_flush enter 2024-07-26T23:34:20.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 10 librados: watch_flush exit 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 20 client.4151.objecter shutdown clearing up homeless session... 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 10 client.4151.objecter successfully canceled tick 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 10 monclient: shutdown 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 1 -- 172.21.15.139:0/981582182 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 msgr2=0x7ff9481d8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff9481d8640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff93800bab0 tx=0x7ff93800bf70 comp rx=0 tx=0).stop 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 1 -- 172.21.15.139:0/981582182 shutdown_connections 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9480b1240 0x7ff9481d8640 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff948140500 0x7ff9481d8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 1 --2- 172.21.15.139:0/981582182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9480aeec0 0x7ff94813ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.364+0000 7ff934ff9640 1 -- 172.21.15.139:0/981582182 >> 172.21.15.139:0/981582182 conn(0x7ff9480b1c10 msgr2=0x7ff94813b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:20.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.365+0000 7ff934ff9640 1 -- 172.21.15.139:0/981582182 shutdown_connections 2024-07-26T23:34:20.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.365+0000 7ff934ff9640 1 -- 172.21.15.139:0/981582182 wait complete. 2024-07-26T23:34:20.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:34:20.365+0000 7ff934ff9640 1 librados: shutdown 2024-07-26T23:34:20.392 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-07-26T23:34:20.393 INFO:tasks.ceph:Waiting for OSDs to come up 2024-07-26T23:34:21.200 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:34:21.408 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.398+0000 7fd960f15640 10 monclient: get_monmap_and_config 2024-07-26T23:34:21.408 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.398+0000 7fd960f15640 10 monclient: build_initial_monmap 2024-07-26T23:34:21.408 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.398+0000 7fd960f15640 10 monclient: monmap: 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:34:21.399225+0000 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:34:21.399225+0000 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:21.409 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:21.432 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.430+0000 7fd960f15640 1 Processor -- start 2024-07-26T23:34:21.443 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.432+0000 7fd960f15640 1 -- start start 2024-07-26T23:34:21.443 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.432+0000 7fd960f15640 10 monclient: init 2024-07-26T23:34:21.449 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:21.449 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-07-26T23:34:21.449 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954034610 0x7fd954030250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:21.449 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient(hunting): picked mon.noname-b con 0x7fd954034610 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:21.449 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd9540349f0 0x7fd954030790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient(hunting): picked mon.noname-c con 0x7fd9540349f0 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd954030cd0 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd95402c790 con 0x7fd954030cd0 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd954032d50 con 0x7fd9540349f0 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fd95402beb0 con 0x7fd954034610 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.433+0000 7fd960f15640 10 monclient(hunting): authenticate will time out at 1294.469446s 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.436+0000 7fd95ad9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fd954030cd0 0x7fd9540312d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:40078/0 (socket says 172.21.15.139:40078) 2024-07-26T23:34:21.450 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.436+0000 7fd95ad9d640 1 -- 172.21.15.139:0/805580178 learned_addr learned my addr 172.21.15.139:0/805580178 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:34:21.451 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.447+0000 7fd959d9b640 1 --2- 172.21.15.139:0/805580178 >> v2:172.21.15.177:3300/0 conn(0x7fd9540349f0 0x7fd954030790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:21.451 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.448+0000 7fd95a59c640 1 --2- 172.21.15.139:0/805580178 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954034610 0x7fd954030250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.449+0000 7fd95a59c640 10 monclient(hunting): get_auth_request con 0x7fd954034610 auth_method 0 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.449+0000 7fd95a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.449+0000 7fd95a59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.449+0000 7fd95a59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95959a640 1 -- 172.21.15.139:0/805580178 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3791274310 0 0) 0x7fd95402c790 con 0x7fd954030cd0 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95959a640 10 monclient(hunting): _negotiate 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95959a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95959a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:21.452 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95959a640 10 monclient(hunting): my global_id is 4165 2024-07-26T23:34:21.453 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95959a640 1 -- 172.21.15.139:0/805580178 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd944003520 con 0x7fd954030cd0 2024-07-26T23:34:21.453 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:21.453 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:21.453 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.450+0000 7fd95a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:21.456 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 10 monclient(hunting): handle_auth_done global_id 4154 payload 274 2024-07-26T23:34:21.456 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 1 -- 172.21.15.139:0/805580178 >> v2:172.21.15.177:3300/0 conn(0x7fd9540349f0 msgr2=0x7fd954030790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:21.456 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 1 --2- 172.21.15.139:0/805580178 >> v2:172.21.15.177:3300/0 conn(0x7fd9540349f0 0x7fd954030790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:21.456 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 1 -- 172.21.15.139:0/805580178 >> v1:172.21.15.177:6789/0 conn(0x7fd954030cd0 legacy=0x7fd9540312d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:21.456 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 10 monclient: _finish_hunting 0 2024-07-26T23:34:21.456 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 1 monclient: found mon.noname-b 2024-07-26T23:34:21.457 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:21.457 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:21.457 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.453+0000 7fd95a59c640 1 -- 172.21.15.139:0/805580178 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95402fa50 con 0x7fd954034610 2024-07-26T23:34:21.458 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.457+0000 7fd95a59c640 1 --2- 172.21.15.139:0/805580178 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954034610 0x7fd954030250 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd9500071d0 tx=0x7fd95002caa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd4d5e7ccfbb1d18 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd95959a640 1 -- 172.21.15.139:0/805580178 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950034070 con 0x7fd954034610 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd95959a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd95959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd95959a640 10 monclient: dump: 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:21.507 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd95959a640 10 monclient: _finish_auth 0 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd95959a640 10 monclient: _check_auth_tickets 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd95959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd960f15640 5 monclient: authenticate success, global_id 4154 2024-07-26T23:34:21.508 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.505+0000 7fd960f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:21.625 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.622+0000 7fd95959a640 1 -- 172.21.15.139:0/805580178 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd950008df0 con 0x7fd954034610 2024-07-26T23:34:21.625 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.622+0000 7fd95959a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:21.625 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.622+0000 7fd960f15640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:21.625 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.622+0000 7fd960f15640 4 set_mon_vals no callback set 2024-07-26T23:34:21.625 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 10 monclient: shutdown 2024-07-26T23:34:21.625 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:21.625 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 1 -- 172.21.15.139:0/805580178 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954034610 msgr2=0x7fd954030250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:21.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd95959a640 1 -- 172.21.15.139:0/805580178 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950034070 con 0x7fd954034610 2024-07-26T23:34:21.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 1 --2- 172.21.15.139:0/805580178 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954034610 0x7fd954030250 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd9500071d0 tx=0x7fd95002caa0 comp rx=0 tx=0).stop 2024-07-26T23:34:21.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd95959a640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-07-26T23:34:21.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 1 -- 172.21.15.139:0/805580178 shutdown_connections 2024-07-26T23:34:21.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 1 --2- 172.21.15.139:0/805580178 >> v2:172.21.15.177:3300/0 conn(0x7fd9540349f0 0x7fd954030790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:21.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 1 --2- 172.21.15.139:0/805580178 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954034610 0x7fd954030250 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:21.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 1 -- 172.21.15.139:0/805580178 >> 172.21.15.139:0/805580178 conn(0x7fd954005c00 msgr2=0x7fd954006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:21.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.623+0000 7fd960f15640 1 -- 172.21.15.139:0/805580178 shutdown_connections 2024-07-26T23:34:21.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.624+0000 7fd960f15640 1 -- 172.21.15.139:0/805580178 wait complete. 2024-07-26T23:34:21.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.625+0000 7fd960f15640 10 monclient: build_initial_monmap 2024-07-26T23:34:21.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.625+0000 7fd960f15640 10 monclient: monmap: 2024-07-26T23:34:21.627 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:21.627 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.625+0000 7fd960f15640 1 librados: starting msgr at 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.625+0000 7fd960f15640 1 librados: starting objecter 2024-07-26T23:34:21.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.625+0000 7fd960f15640 1 Processor -- start 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.626+0000 7fd960f15640 1 -- start start 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.626+0000 7fd960f15640 1 librados: setting wanted keys 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.626+0000 7fd960f15640 1 librados: calling monclient init 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.626+0000 7fd960f15640 10 monclient: init 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.627+0000 7fd960f15640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.627+0000 7fd960f15640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.627+0000 7fd960f15640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954030cd0 0x7fd9540c4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.627+0000 7fd960f15640 10 monclient(hunting): picked mon.noname-c con 0x7fd954030cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:21.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.627+0000 7fd960f15640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd954034610 0x7fd9540c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:21.631 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.627+0000 7fd960f15640 10 monclient(hunting): picked mon.noname-a con 0x7fd954034610 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:21.631 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.627+0000 7fd960f15640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9540349f0 0x7fd9540c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:21.632 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.629+0000 7fd959d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd954034610 0x7fd9540c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:21.632 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd959d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd954034610 0x7fd9540c6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:45744/0 (socket says 172.21.15.139:45744) 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 10 monclient(hunting): picked mon.noname-b con 0x7fd9540349f0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd959d9b640 1 -- 172.21.15.139:0/2996961600 learned_addr learned my addr 172.21.15.139:0/2996961600 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd9540319e0 con 0x7fd954034610 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd954006930 con 0x7fd9540349f0 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fd954031b60 con 0x7fd954030cd0 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.630+0000 7fd960f15640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:21.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.631+0000 7fd960f15640 10 monclient(hunting): authenticate will time out at 1294.667052s 2024-07-26T23:34:21.634 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.631+0000 7fd959d9b640 10 monclient(hunting): get_auth_request con 0x7fd954034610 auth_method 0 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.631+0000 7fd959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.631+0000 7fd959d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.631+0000 7fd959d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd95ad9d640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9540349f0 0x7fd9540c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd95ad9d640 10 monclient(hunting): get_auth_request con 0x7fd9540349f0 auth_method 0 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd95ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd95ad9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:21.635 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd95ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:21.641 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd959d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd959d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.632+0000 7fd959d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.638+0000 7fd95ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.638+0000 7fd95ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.638+0000 7fd95ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 10 monclient(hunting): handle_auth_done global_id 4170 payload 824 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 1 -- 172.21.15.139:0/2996961600 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954030cd0 msgr2=0x7fd9540c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954030cd0 0x7fd9540c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 1 -- 172.21.15.139:0/2996961600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9540349f0 msgr2=0x7fd9540c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:21.642 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9540349f0 0x7fd9540c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:21.643 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 10 monclient: _finish_hunting 0 2024-07-26T23:34:21.643 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 1 monclient: found mon.noname-a 2024-07-26T23:34:21.643 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:21.643 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:34:21.643 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.639+0000 7fd959d9b640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9540c4aa0 con 0x7fd954034610 2024-07-26T23:34:21.644 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.643+0000 7fd959d9b640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd954034610 0x7fd9540c6d80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00c850 tx=0x7fd94c00cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:21.644 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.643+0000 7fd94affd640 10 client.?.objecter ms_handle_connect 0x7fd954034610 2024-07-26T23:34:21.644 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.643+0000 7fd94affd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd94affd640 1 -- 172.21.15.139:0/2996961600 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94c015070 con 0x7fd954034610 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd94affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd94affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd94affd640 10 monclient: dump: 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:21.766 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd94affd640 10 monclient: _finish_auth 0 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd94affd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd94affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd960f15640 5 monclient: authenticate success, global_id 4170 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd960f15640 10 monclient: _renew_subs 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.763+0000 7fd960f15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:21.767 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.764+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9540c57d0 con 0x7fd954034610 2024-07-26T23:34:21.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.764+0000 7fd960f15640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:21.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.764+0000 7fd960f15640 10 monclient: _renew_subs 2024-07-26T23:34:21.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.764+0000 7fd960f15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:21.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.764+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9540c7800 con 0x7fd954034610 2024-07-26T23:34:21.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.764+0000 7fd960f15640 1 librados: init done 2024-07-26T23:34:21.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.765+0000 7fd960f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.765+0000 7fd960f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.765+0000 7fd960f15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.765+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd954031f70 con 0x7fd954034610 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.766+0000 7fd94affd640 1 -- 172.21.15.139:0/2996961600 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd94c004220 con 0x7fd954034610 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.766+0000 7fd94affd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.766+0000 7fd94affd640 1 -- 172.21.15.139:0/2996961600 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94c01e060 con 0x7fd954034610 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.766+0000 7fd94affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.766+0000 7fd958d99640 4 set_mon_vals no callback set 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.766+0000 7fd94affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.766+0000 7fd94affd640 10 monclient: dump: 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:21.769 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:21.770 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:21.950 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.948+0000 7fd94affd640 1 -- 172.21.15.139:0/2996961600 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 2) ==== 49591+0+0 (secure 0 0 0) 0x7fd94c007620 con 0x7fd954034610 2024-07-26T23:34:21.959 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 1 -- 172.21.15.139:0/2996961600 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fd94c007d40 con 0x7fd954034610 2024-07-26T23:34:21.959 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 10 client.4170.objecter ms_dispatch 0x7fd95c000b90 osd_map(10..10 src has 1..10) 2024-07-26T23:34:21.959 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 3 client.4170.objecter handle_osd_map got epochs [10,10] > 0 2024-07-26T23:34:21.959 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 3 client.4170.objecter handle_osd_map decoding full epoch 10 2024-07-26T23:34:21.959 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 20 client.4170.objecter dump_active .. 0 homeless 2024-07-26T23:34:21.959 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 1 -- 172.21.15.139:0/2996961600 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74442 (secure 0 0 0) 0x7fd94c003cf0 con 0x7fd954034610 2024-07-26T23:34:21.960 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:21.960 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:21.957+0000 7fd94affd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:22.090 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.088+0000 7fd960f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:22.090 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.088+0000 7fd960f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:22.090 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.088+0000 7fd960f15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:22.090 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.088+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd95402c4d0 con 0x7fd954034610 2024-07-26T23:34:22.156 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.154+0000 7fd94affd640 1 -- 172.21.15.139:0/2996961600 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fd94c003990 con 0x7fd954034610 2024-07-26T23:34:22.156 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.154+0000 7fd94affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:22.157 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.154+0000 7fd94affd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:22.157 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:34:22.158 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":10,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:34:19.494450+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:34:22.160 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 10 librados: watch_flush enter 2024-07-26T23:34:22.160 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 10 librados: watch_flush exit 2024-07-26T23:34:22.161 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 20 client.4170.objecter shutdown clearing up homeless session... 2024-07-26T23:34:22.161 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 10 client.4170.objecter successfully canceled tick 2024-07-26T23:34:22.161 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 10 monclient: shutdown 2024-07-26T23:34:22.161 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:22.161 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd954034610 msgr2=0x7fd9540c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:22.162 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd954034610 0x7fd9540c6d80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00c850 tx=0x7fd94c00cd10 comp rx=0 tx=0).stop 2024-07-26T23:34:22.162 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 shutdown_connections 2024-07-26T23:34:22.162 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9540349f0 0x7fd9540c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:22.162 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd954034610 0x7fd9540c6d80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:22.162 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 --2- 172.21.15.139:0/2996961600 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd954030cd0 0x7fd9540c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:22.163 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 >> 172.21.15.139:0/2996961600 conn(0x7fd954005c00 msgr2=0x7fd9540c09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:22.163 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.156+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 shutdown_connections 2024-07-26T23:34:22.163 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.157+0000 7fd960f15640 1 -- 172.21.15.139:0/2996961600 wait complete. 2024-07-26T23:34:22.163 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:22.157+0000 7fd960f15640 1 librados: shutdown 2024-07-26T23:34:22.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-26T23:34:23.255 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-07-26T23:34:23.252+0000 7f6fd2cf6640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-07-26T23:34:23.299 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-07-26T23:34:23.298+0000 7f6fd2cf6640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-07-26T23:34:27.090 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:07.387 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.098 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:07.390 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.154 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:07.445 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.163 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:07.455 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.174 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:07.471 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.215 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:07.506 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.345 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:07.642 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.407 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:07.704 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.708 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-07-26T23:34:27.699+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:27.741 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-07-26T23:34:27.734+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:27.797 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-07-26T23:34:27.789+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:27.938 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:08.217 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:27.962 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:34:27.955+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:28.005 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:08.284 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.090 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:08.373 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.149 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:08.434 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.204 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:08.501 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.209 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:08.480 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.248 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:08.541 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.270 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:08.540 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.358 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:08.650 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.384 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:08.660 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.443 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:08.719 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.525 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:08.821 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.543 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:34:28.536+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:28.598 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:08.894 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.641 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:08.934 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.716 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:34:28.709+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:28.745 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.037 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.838 INFO:tasks.ceph.osd.3.smithi139.stderr:2024-07-26T23:34:28.830+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:28.891 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.187 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.913 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.210 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.937 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.229 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:28.978 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:34:29.031 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:34:29.022+0000 5dce500 -1 Falling back to public interface 2024-07-26T23:34:29.036 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:09.316 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.041 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.332 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.108 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.107+0000 7efd7668e640 10 monclient: get_monmap_and_config 2024-07-26T23:34:29.108 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.107+0000 7efd7668e640 10 monclient: build_initial_monmap 2024-07-26T23:34:29.108 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.107+0000 7efd7668e640 10 monclient: monmap: 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:34:29.108163+0000 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:34:29.108163+0000 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:29.109 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:29.110 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:29.110 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:29.112 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.107+0000 7efd7668e640 1 Processor -- start 2024-07-26T23:34:29.112 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.107+0000 7efd7668e640 1 -- start start 2024-07-26T23:34:29.112 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.107+0000 7efd7668e640 10 monclient: init 2024-07-26T23:34:29.112 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:29.112 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-07-26T23:34:29.112 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd700b62c0 0x7efd70143970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient(hunting): picked mon.noname-c con 0x7efd700b62c0 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd700b66a0 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd70144490 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd700acc10 con 0x7efd700b66a0 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd7013a500 con 0x7efd70144490 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7efd70144920 con 0x7efd700b62c0 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.108+0000 7efd7668e640 10 monclient(hunting): authenticate will time out at 1302.144695s 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.109+0000 7efd6f7fe640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7efd700b66a0 0x7efd700b4250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:49522/0 (socket says 172.21.15.139:49522) 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.109+0000 7efd6f7fe640 1 -- 172.21.15.139:0/212815586 learned_addr learned my addr 172.21.15.139:0/212815586 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:34:29.113 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.109+0000 7efd6ffff640 1 --2- 172.21.15.139:0/212815586 >> v2:172.21.15.177:3300/0 conn(0x7efd700b62c0 0x7efd70143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:29.114 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.110+0000 7efd6ffff640 10 monclient(hunting): get_auth_request con 0x7efd700b62c0 auth_method 0 2024-07-26T23:34:29.114 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.110+0000 7efd6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:29.114 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.110+0000 7efd6ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:29.114 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.110+0000 7efd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:29.114 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.110+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:29.114 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.110+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:29.114 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.110+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:29.115 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.111+0000 7efd6effd640 1 -- 172.21.15.139:0/212815586 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2215666442 0 0) 0x7efd700acc10 con 0x7efd700b66a0 2024-07-26T23:34:29.115 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.111+0000 7efd6effd640 10 monclient(hunting): _negotiate 2024-07-26T23:34:29.115 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.111+0000 7efd6effd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:29.115 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.111+0000 7efd6effd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:29.115 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.111+0000 7efd6effd640 10 monclient(hunting): my global_id is 4203 2024-07-26T23:34:29.115 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.111+0000 7efd6effd640 1 -- 172.21.15.139:0/212815586 --> v1:172.21.15.139:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd58003520 con 0x7efd700b66a0 2024-07-26T23:34:29.115 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 10 monclient(hunting): handle_auth_done global_id 4195 payload 274 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 1 -- 172.21.15.139:0/212815586 >> v1:172.21.15.177:6789/0 conn(0x7efd70144490 legacy=0x7efd70139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 1 -- 172.21.15.139:0/212815586 >> v1:172.21.15.139:6789/0 conn(0x7efd700b66a0 legacy=0x7efd700b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 1 monclient: found mon.noname-c 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.112+0000 7efd6ffff640 1 -- 172.21.15.139:0/212815586 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7013a6d0 con 0x7efd700b62c0 2024-07-26T23:34:29.116 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.115+0000 7efd6ffff640 1 --2- 172.21.15.139:0/212815586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd70143970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd600097a0 tx=0x7efd6002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c9ca561ade9b5cd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:29.126 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 1 -- 172.21.15.139:0/212815586 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600315a0 con 0x7efd700b62c0 2024-07-26T23:34:29.126 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:29.126 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:29.126 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: dump: 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:29.127 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: _finish_auth 0 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 1 -- 172.21.15.139:0/212815586 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60037070 con 0x7efd700b62c0 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 1 -- 172.21.15.139:0/212815586 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60031960 con 0x7efd700b62c0 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd6effd640 10 monclient: dump: 2024-07-26T23:34:29.128 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:29.129 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 5 monclient: authenticate success, global_id 4195 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 4 set_mon_vals no callback set 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 10 monclient: shutdown 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 1 -- 172.21.15.139:0/212815586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 msgr2=0x7efd70143970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 1 --2- 172.21.15.139:0/212815586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd70143970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd600097a0 tx=0x7efd6002b840 comp rx=0 tx=0).stop 2024-07-26T23:34:29.130 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 1 -- 172.21.15.139:0/212815586 shutdown_connections 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.119+0000 7efd7668e640 1 --2- 172.21.15.139:0/212815586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd70143970 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.120+0000 7efd7668e640 1 -- 172.21.15.139:0/212815586 >> 172.21.15.139:0/212815586 conn(0x7efd7006e710 msgr2=0x7efd7002d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.120+0000 7efd7668e640 1 -- 172.21.15.139:0/212815586 shutdown_connections 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.120+0000 7efd7668e640 1 -- 172.21.15.139:0/212815586 wait complete. 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.120+0000 7efd7668e640 10 monclient: build_initial_monmap 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.120+0000 7efd7668e640 10 monclient: monmap: 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:29.131 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.120+0000 7efd7668e640 1 librados: starting msgr at 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.120+0000 7efd7668e640 1 librados: starting objecter 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 Processor -- start 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 -- start start 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 librados: setting wanted keys 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 librados: calling monclient init 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient: init 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:34:29.132 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd701468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): picked mon.noname-b con 0x7efd700b62c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7efd700b66a0 0x7efd7013fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): picked mon.noname-c con 0x7efd700b66a0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd70144490 0x7efd701dc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): picked mon.noname-a con 0x7efd70144490 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7efd700b6ac0 con 0x7efd70144490 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efd700b4960 con 0x7efd700b62c0 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7efd7014f4a0 con 0x7efd700b66a0 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.121+0000 7efd7668e640 10 monclient(hunting): authenticate will time out at 1302.157904s 2024-07-26T23:34:29.133 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6f7fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7efd700b66a0 0x7efd7013fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd701468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd74c2b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd70144490 0x7efd701dc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd701468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:38544/0 (socket says 172.21.15.139:38544) 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6ffff640 1 -- 172.21.15.139:0/2396583729 learned_addr learned my addr 172.21.15.139:0/2396583729 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6ffff640 10 monclient(hunting): get_auth_request con 0x7efd700b62c0 auth_method 0 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6f7fe640 10 monclient(hunting): get_auth_request con 0x7efd700b66a0 auth_method 0 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:29.134 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd74c2b640 10 monclient(hunting): get_auth_request con 0x7efd70144490 auth_method 0 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd74c2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd74c2b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.122+0000 7efd74c2b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd74c2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd74c2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:29.135 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.123+0000 7efd74c2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:29.138 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 10 monclient(hunting): handle_auth_done global_id 4198 payload 824 2024-07-26T23:34:29.138 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 1 -- 172.21.15.139:0/2396583729 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7efd700b66a0 msgr2=0x7efd7013fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7efd700b66a0 0x7efd7013fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 1 -- 172.21.15.139:0/2396583729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd70144490 msgr2=0x7efd701dc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd70144490 0x7efd701dc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 1 monclient: found mon.noname-b 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.125+0000 7efd6ffff640 1 -- 172.21.15.139:0/2396583729 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7013ffa0 con 0x7efd700b62c0 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.128+0000 7efd6ffff640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd701468b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efd60034040 tx=0x7efd600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.128+0000 7efd6cff9640 10 client.?.objecter ms_handle_connect 0x7efd700b62c0 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.128+0000 7efd6cff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:29.139 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 1 -- 172.21.15.139:0/2396583729 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60032630 con 0x7efd700b62c0 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 10 monclient: dump: 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:29.140 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 10 monclient: _finish_auth 0 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 10 monclient: _check_auth_tickets 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd7668e640 5 monclient: authenticate success, global_id 4198 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 1 -- 172.21.15.139:0/2396583729 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd600327d0 con 0x7efd700b62c0 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6cff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.131+0000 7efd6e7fc640 4 set_mon_vals no callback set 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd6cff9640 1 -- 172.21.15.139:0/2396583729 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60032bc0 con 0x7efd700b62c0 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd6cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd6cff9640 10 monclient: dump: 2024-07-26T23:34:29.141 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 10 monclient: _renew_subs 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd70148380 con 0x7efd700b62c0 2024-07-26T23:34:29.142 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 10 monclient: _renew_subs 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd70148040 con 0x7efd700b62c0 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.132+0000 7efd7668e640 1 librados: init done 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.133+0000 7efd7668e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.133+0000 7efd7668e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.133+0000 7efd7668e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:29.143 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.133+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd70143970 con 0x7efd700b62c0 2024-07-26T23:34:29.155 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.452 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.213 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.505 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.256 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.552 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.263 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:09.548 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.305 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.597 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.389 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.387+0000 7efd6cff9640 1 -- 172.21.15.139:0/2396583729 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 3) ==== 99083+0+0 (secure 0 0 0) 0x7efd6004e430 con 0x7efd700b62c0 2024-07-26T23:34:29.389 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.388+0000 7efd6cff9640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7efd5c078960 0x7efd5c07ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:29.389 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.388+0000 7efd6f7fe640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7efd5c078960 0x7efd5c07ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:29.390 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.388+0000 7efd6f7fe640 10 monclient: get_auth_request con 0x7efd5c078960 auth_method 0 2024-07-26T23:34:29.390 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.389+0000 7efd6cff9640 10 client.4198.objecter ms_handle_connect 0x7efd5c078960 2024-07-26T23:34:29.390 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.389+0000 7efd6f7fe640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7efd5c078960 0x7efd5c07ae20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efd70140a40 tx=0x7efd64007cd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:29.393 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.392+0000 7efd6cff9640 1 -- 172.21.15.139:0/2396583729 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7efd6002f030 con 0x7efd700b62c0 2024-07-26T23:34:29.393 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.392+0000 7efd6cff9640 10 client.4198.objecter ms_dispatch 0x7efd70000b90 osd_map(10..10 src has 1..10) 2024-07-26T23:34:29.394 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.392+0000 7efd6cff9640 3 client.4198.objecter handle_osd_map got epochs [10,10] > 0 2024-07-26T23:34:29.394 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.392+0000 7efd6cff9640 3 client.4198.objecter handle_osd_map decoding full epoch 10 2024-07-26T23:34:29.394 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.392+0000 7efd6cff9640 20 client.4198.objecter dump_active .. 0 homeless 2024-07-26T23:34:29.402 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:09.673 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.405 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.404+0000 7efd6cff9640 1 -- 172.21.15.139:0/2396583729 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7efd60040070 con 0x7efd700b62c0 2024-07-26T23:34:29.405 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.404+0000 7efd6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:29.405 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.404+0000 7efd6cff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:29.422 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.719 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.481 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:09.761 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.482 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:09.773 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.548 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:09.845 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.559 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:09.835 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.573 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:09.865 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.629+0000 7efd7668e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:29.631 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.629+0000 7efd7668e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:29.631 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.629+0000 7efd7668e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:29.631 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.629+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd70147ab0 con 0x7efd700b62c0 2024-07-26T23:34:29.669 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:09.954 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.673 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.672+0000 7efd6cff9640 1 -- 172.21.15.139:0/2396583729 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7efd600833a0 con 0x7efd700b62c0 2024-07-26T23:34:29.674 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.672+0000 7efd6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:29.674 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.672+0000 7efd6cff9640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:29.674 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:34:29.674 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":10,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:34:19.494450+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:34:29.676 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 10 librados: watch_flush enter 2024-07-26T23:34:29.676 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 10 librados: watch_flush exit 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 20 client.4198.objecter shutdown clearing up homeless session... 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 10 client.4198.objecter successfully canceled tick 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7efd5c078960 msgr2=0x7efd5c07ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7efd5c078960 0x7efd5c07ae20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efd70140a40 tx=0x7efd64007cd0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 10 monclient: shutdown 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 msgr2=0x7efd701468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd701468b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efd60034040 tx=0x7efd600046c0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 shutdown_connections 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7efd5c078960 0x7efd5c07ae20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7efd700b66a0 0x7efd7013fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd70144490 0x7efd701dc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 --2- 172.21.15.139:0/2396583729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd700b62c0 0x7efd701468b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:29.678 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.674+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 >> 172.21.15.139:0/2396583729 conn(0x7efd7006e710 msgr2=0x7efd700b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:29.678 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.675+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 shutdown_connections 2024-07-26T23:34:29.678 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.675+0000 7efd7668e640 1 -- 172.21.15.139:0/2396583729 wait complete. 2024-07-26T23:34:29.678 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:29.675+0000 7efd7668e640 1 librados: shutdown 2024-07-26T23:34:29.686 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-26T23:34:29.690 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:09.987 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.750 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.042 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.827 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:10.106 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.846 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:10.117 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.851 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.148 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.856 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.147 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.954 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.251 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:29.958 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:10.234 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.007 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:10.292 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.019 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.312 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.115 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:10.395 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.132 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.423 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.149 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.446 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.187 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:10.458 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.219 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.515 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.284 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.576 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.292 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:10.568 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.294 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:10.579 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.398 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.688 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.413 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:10.693 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.436 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:10.733 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.476 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:10.747 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.482 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:10.779 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.547 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:10.839 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.583 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:10.859 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.587 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:10.872 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.664 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:10.956 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.700 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:10.980 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.731 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.028 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.763 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:11.034 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.879 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:11.155 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.884 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:11.169 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.983 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.279 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:30.987 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:11.267 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.014 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:11.311 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.031 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.328 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.054 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:11.325 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.063 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.355 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.097 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:11.387 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.161 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:11.452 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.174 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:11.450 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.178 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:11.463 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.193 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:11.484 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.277 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:11.557 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.341 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:11.612 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.471 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:11.748 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.479 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:11.764 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.571 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:11.851 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.603 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.900 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.632 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:11.903 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.634 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:11.931 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.760 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:12.035 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.775 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:12.060 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:31.935 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:12.206 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.072 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:12.347 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.122 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:12.401 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.152 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:12.432 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.332 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:12.617 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.364 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:12.649 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.505 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:12.775 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.551 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:12.822 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.673 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:12.949 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:32.705 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:12.981 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:36.459 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:16.755 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:36.483 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:16.775 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:36.495 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:34:36.565 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:16.856 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.591+0000 7f49e9095640 10 monclient: get_monmap_and_config 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.591+0000 7f49e9095640 10 monclient: build_initial_monmap 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.591+0000 7f49e9095640 10 monclient: monmap: 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:34:36.591474+0000 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:34:36.591474+0000 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:36.592 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:36.593 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:36.593 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:36.593 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:36.594 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 1 Processor -- start 2024-07-26T23:34:36.594 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 1 -- start start 2024-07-26T23:34:36.594 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 10 monclient: init 2024-07-26T23:34:36.594 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:36.594 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49e40b62c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49e4143970 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.592+0000 7f49e9095640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f49e413a500 0x7f49e40b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e9095640 10 monclient(hunting): picked mon.noname-c con 0x7f49e413a500 addr v2:172.21.15.177:3300/0 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e9095640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49e40acc10 con 0x7f49e40b62c0 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e9095640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49e4144490 con 0x7f49e4143970 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e9095640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e9095640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f49e40b4960 con 0x7f49e413a500 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e9095640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e9095640 10 monclient(hunting): authenticate will time out at 1309.628099s 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e359e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f49e413a500 0x7f49e40b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e2d9d640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f49e40b62c0 0x7f49e40b4250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:45044/0 (socket says 172.21.15.139:45044) 2024-07-26T23:34:36.595 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.593+0000 7f49e2d9d640 1 -- 172.21.15.139:0/3986685298 learned_addr learned my addr 172.21.15.139:0/3986685298 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:34:36.596 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.594+0000 7f49e359e640 10 monclient(hunting): get_auth_request con 0x7f49e413a500 auth_method 0 2024-07-26T23:34:36.596 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.594+0000 7f49e359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:36.596 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.594+0000 7f49e359e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:36.596 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.594+0000 7f49e359e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:36.596 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.594+0000 7f49e359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.594+0000 7f49e359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.594+0000 7f49e359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 10 monclient(hunting): handle_auth_done global_id 4201 payload 274 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 1 -- 172.21.15.139:0/3986685298 >> v1:172.21.15.177:6789/0 conn(0x7f49e4143970 legacy=0x7f49e4139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 1 -- 172.21.15.139:0/3986685298 >> v1:172.21.15.139:6789/0 conn(0x7f49e40b62c0 legacy=0x7f49e40b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 10 monclient: _finish_hunting 0 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 1 monclient: found mon.noname-c 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:34:36.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.596+0000 7f49e359e640 1 -- 172.21.15.139:0/3986685298 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e4144660 con 0x7f49e413a500 2024-07-26T23:34:36.601 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.601+0000 7f49e359e640 1 --2- 172.21.15.139:0/3986685298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 0x7f49e40b66a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009d90 tx=0x7f49dc02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d00e93bf99ed766 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:36.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e1d9b640 1 -- 172.21.15.139:0/3986685298 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc039070 con 0x7f49e413a500 2024-07-26T23:34:36.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:36.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:36.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e1d9b640 10 monclient: dump: 2024-07-26T23:34:36.605 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:36.605 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:36.605 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e1d9b640 10 monclient: _finish_auth 0 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e1d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e9095640 5 monclient: authenticate success, global_id 4201 2024-07-26T23:34:36.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.604+0000 7f49e9095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:36.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e1d9b640 1 -- 172.21.15.139:0/3986685298 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49dc02cb30 con 0x7f49e413a500 2024-07-26T23:34:36.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e1d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:36.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e1d9b640 1 -- 172.21.15.139:0/3986685298 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc034440 con 0x7f49e413a500 2024-07-26T23:34:36.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e9095640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:36.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e9095640 4 set_mon_vals no callback set 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e1d9b640 10 monclient: dump: 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:36.608 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:36.609 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e9095640 10 monclient: shutdown 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e9095640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e9095640 1 -- 172.21.15.139:0/3986685298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 msgr2=0x7f49e40b66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.607+0000 7f49e9095640 1 --2- 172.21.15.139:0/3986685298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 0x7f49e40b66a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009d90 tx=0x7f49dc02bd80 comp rx=0 tx=0).stop 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.608+0000 7f49e9095640 1 -- 172.21.15.139:0/3986685298 shutdown_connections 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.608+0000 7f49e9095640 1 --2- 172.21.15.139:0/3986685298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 0x7f49e40b66a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.608+0000 7f49e9095640 1 -- 172.21.15.139:0/3986685298 >> 172.21.15.139:0/3986685298 conn(0x7f49e406e710 msgr2=0x7f49e402d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.608+0000 7f49e9095640 1 -- 172.21.15.139:0/3986685298 shutdown_connections 2024-07-26T23:34:36.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.608+0000 7f49e9095640 1 -- 172.21.15.139:0/3986685298 wait complete. 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 10 monclient: build_initial_monmap 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 10 monclient: monmap: 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:36.611 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:36.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 1 librados: starting msgr at 2024-07-26T23:34:36.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 1 librados: starting objecter 2024-07-26T23:34:36.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 1 Processor -- start 2024-07-26T23:34:36.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 1 -- start start 2024-07-26T23:34:36.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 1 librados: setting wanted keys 2024-07-26T23:34:36.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 1 librados: calling monclient init 2024-07-26T23:34:36.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.609+0000 7f49e9095640 10 monclient: init 2024-07-26T23:34:36.614 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49e40b62c0 0x7f49e4147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): picked mon.noname-a con 0x7f49e40b62c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 0x7f49e413e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): picked mon.noname-b con 0x7f49e413a500 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f49e4143970 0x7f49e4144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): picked mon.noname-c con 0x7f49e4143970 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f49e414f620 con 0x7f49e40b62c0 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49e414f4a0 con 0x7f49e413a500 2024-07-26T23:34:36.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f49e414f7a0 con 0x7f49e4143970 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.610+0000 7f49e9095640 10 monclient(hunting): authenticate will time out at 1309.645936s 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e2d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49e40b62c0 0x7f49e4147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 0x7f49e413e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e359e640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f49e4143970 0x7f49e4144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e359e640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f49e4143970 0x7f49e4144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48002/0 (socket says 172.21.15.139:48002) 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e359e640 1 -- 172.21.15.139:0/3905577167 learned_addr learned my addr 172.21.15.139:0/3905577167 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e359e640 10 monclient(hunting): get_auth_request con 0x7f49e4143970 auth_method 0 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e359e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.611+0000 7f49e359e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:36.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e2d9d640 10 monclient(hunting): get_auth_request con 0x7f49e40b62c0 auth_method 0 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e2d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e259c640 10 monclient(hunting): get_auth_request con 0x7f49e413a500 auth_method 0 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.612+0000 7f49e359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.613+0000 7f49e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.613+0000 7f49e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:36.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.613+0000 7f49e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:36.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.613+0000 7f49e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:36.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.613+0000 7f49e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:36.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.613+0000 7f49e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 1 -- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 msgr2=0x7f49e413e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 0x7f49e413e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 1 -- 172.21.15.139:0/3905577167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49e40b62c0 msgr2=0x7f49e4147930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49e40b62c0 0x7f49e4147930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 10 monclient: _finish_hunting 0 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 1 monclient: found mon.noname-c 2024-07-26T23:34:36.619 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:36.620 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:34:36.620 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.616+0000 7f49e359e640 1 -- 172.21.15.139:0/3905577167 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e413ee40 con 0x7f49e4143970 2024-07-26T23:34:36.620 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.620+0000 7f49e359e640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f49e4143970 0x7f49e4144f70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009eb0 tx=0x7f49dc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:36.620 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.620+0000 7f49cb7fe640 10 client.?.objecter ms_handle_connect 0x7f49e4143970 2024-07-26T23:34:36.620 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.620+0000 7f49cb7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:36.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.623+0000 7f49cb7fe640 1 -- 172.21.15.139:0/3905577167 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc050070 con 0x7f49e4143970 2024-07-26T23:34:36.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.623+0000 7f49cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:36.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.623+0000 7f49cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:36.626 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.623+0000 7f49cb7fe640 10 monclient: dump: 2024-07-26T23:34:36.626 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:36.626 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 10 monclient: _finish_auth 0 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 1 -- 172.21.15.139:0/3905577167 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49dc034b70 con 0x7f49e4143970 2024-07-26T23:34:36.627 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 1 -- 172.21.15.139:0/3905577167 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc04b420 con 0x7f49e4143970 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 5 monclient: authenticate success, global_id 4178 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 10 monclient: _renew_subs 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e413fba0 con 0x7f49e4143970 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e159a640 4 set_mon_vals no callback set 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49cb7fe640 10 monclient: dump: 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.628 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 10 monclient: _renew_subs 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e4136b60 con 0x7f49e4143970 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.624+0000 7f49e9095640 1 librados: init done 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.625+0000 7f49e9095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.625+0000 7f49e9095640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:36.629 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.625+0000 7f49e9095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:36.630 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.625+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49e40b66a0 con 0x7f49e4143970 2024-07-26T23:34:36.896 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.896+0000 7f49cb7fe640 1 -- 172.21.15.139:0/3905577167 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 3) ==== 99083+0+0 (secure 0 0 0) 0x7f49dc053070 con 0x7f49e4143970 2024-07-26T23:34:36.897 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.897+0000 7f49cb7fe640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f49d0078960 0x7f49d007ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:36.897 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.897+0000 7f49e2d9d640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f49d0078960 0x7f49d007ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:36.897 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.897+0000 7f49e2d9d640 10 monclient: get_auth_request con 0x7f49d0078960 auth_method 0 2024-07-26T23:34:36.898 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.898+0000 7f49cb7fe640 10 client.4178.objecter ms_handle_connect 0x7f49d0078960 2024-07-26T23:34:36.898 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.898+0000 7f49e2d9d640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f49d0078960 0x7f49d007ae20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f49d4000f60 tx=0x7f49d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:36.990 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 1 -- 172.21.15.139:0/3905577167 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f49dc08a980 con 0x7f49e4143970 2024-07-26T23:34:36.990 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 10 client.4178.objecter ms_dispatch 0x7f49e4000b90 osd_map(10..10 src has 1..10) 2024-07-26T23:34:36.990 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2024-07-26T23:34:36.990 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2024-07-26T23:34:36.991 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 20 client.4178.objecter dump_active .. 0 homeless 2024-07-26T23:34:36.991 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 1 -- 172.21.15.139:0/3905577167 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f49dc05a020 con 0x7f49e4143970 2024-07-26T23:34:36.991 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:36.991 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:36.989+0000 7f49cb7fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:37.142 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:17.438 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:37.288 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.287+0000 7f49e9095640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:37.288 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.287+0000 7f49e9095640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:37.288 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.287+0000 7f49e9095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:37.288 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.287+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49e4076800 con 0x7f49e4143970 2024-07-26T23:34:37.301 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.300+0000 7f49cb7fe640 1 -- 172.21.15.139:0/3905577167 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f49e413ee40 con 0x7f49e4143970 2024-07-26T23:34:37.301 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.300+0000 7f49cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:37.301 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.300+0000 7f49cb7fe640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:37.301 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:34:37.302 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":10,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:34:19.494450+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:34:37.303 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.302+0000 7f49e9095640 10 librados: watch_flush enter 2024-07-26T23:34:37.303 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.302+0000 7f49e9095640 10 librados: watch_flush exit 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.302+0000 7f49e9095640 20 client.4178.objecter shutdown clearing up homeless session... 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.302+0000 7f49e9095640 10 client.4178.objecter successfully canceled tick 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f49d0078960 msgr2=0x7f49d007ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f49d0078960 0x7f49d007ae20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f49d4000f60 tx=0x7f49d4009040 comp rx=0 tx=0).stop 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 10 monclient: shutdown 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f49e4143970 msgr2=0x7f49e4144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f49e4143970 0x7f49e4144f70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009eb0 tx=0x7f49dc004a40 comp rx=0 tx=0).stop 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 shutdown_connections 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f49d0078960 0x7f49d007ae20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f49e4143970 0x7f49e4144f70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:37.304 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49e413a500 0x7f49e413e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:37.305 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 --2- 172.21.15.139:0/3905577167 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49e40b62c0 0x7f49e4147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:37.305 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 >> 172.21.15.139:0/3905577167 conn(0x7f49e406e710 msgr2=0x7f49e40b9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:37.305 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 shutdown_connections 2024-07-26T23:34:37.305 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 -- 172.21.15.139:0/3905577167 wait complete. 2024-07-26T23:34:37.305 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:37.303+0000 7f49e9095640 1 librados: shutdown 2024-07-26T23:34:37.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-26T23:34:37.877 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:18.156 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:38.060 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:18.344 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:38.572 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:18.848 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:38.758 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:19.029 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:41.960 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:22.257 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:41.964 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:22.261 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:42.056 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:22.348 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:42.060 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:22.352 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:42.621 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:22.913 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:42.625 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:22.917 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:42.756 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:23.053 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:42.762 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:23.059 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:43.726 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:24.005 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:43.730 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:24.009 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:44.124 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:34:44.216 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.214+0000 7ff99047c640 10 monclient: get_monmap_and_config 2024-07-26T23:34:44.216 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.214+0000 7ff99047c640 10 monclient: build_initial_monmap 2024-07-26T23:34:44.216 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.214+0000 7ff99047c640 10 monclient: monmap: 2024-07-26T23:34:44.216 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:44.216 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:44.216 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:34:44.215936+0000 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:34:44.215936+0000 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:44.217 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:44.219 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.215+0000 7ff99047c640 1 Processor -- start 2024-07-26T23:34:44.219 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.215+0000 7ff99047c640 1 -- start start 2024-07-26T23:34:44.219 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.215+0000 7ff99047c640 10 monclient: init 2024-07-26T23:34:44.219 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff98806e580 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9881444b0 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff98814f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient(hunting): picked mon.noname-b con 0x7ff9880b3a80 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98813a750 con 0x7ff98806e580 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9880ac800 con 0x7ff9881444b0 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff98806e960 con 0x7ff9880b3a80 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff99047c640 10 monclient(hunting): authenticate will time out at 1317.252413s 2024-07-26T23:34:44.220 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.216+0000 7ff98ea19640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff98814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98e218640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7ff98806e580 0x7ff9880b4250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:40746/0 (socket says 172.21.15.139:40746) 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98e218640 1 -- 172.21.15.139:0/365170173 learned_addr learned my addr 172.21.15.139:0/365170173 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98ea19640 10 monclient(hunting): get_auth_request con 0x7ff9880b3a80 auth_method 0 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98ea19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98ea19640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98ea19640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98ea19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98ea19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:44.221 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.217+0000 7ff98ea19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 1 -- 172.21.15.139:0/365170173 >> v1:172.21.15.177:6789/0 conn(0x7ff9881444b0 legacy=0x7ff9880b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 1 -- 172.21.15.139:0/365170173 >> v1:172.21.15.139:6789/0 conn(0x7ff98806e580 legacy=0x7ff9880b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 10 monclient: _finish_hunting 0 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 1 monclient: found mon.noname-b 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98d216640 1 -- 172.21.15.139:0/365170173 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3146344261 0 0) 0x7ff98813a750 con 0x7ff98806e580 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:44.223 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:44.224 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98ea19640 1 -- 172.21.15.139:0/365170173 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9880acae0 con 0x7ff9880b3a80 2024-07-26T23:34:44.224 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.219+0000 7ff98d216640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-07-26T23:34:44.224 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.221+0000 7ff98ea19640 1 --2- 172.21.15.139:0/365170173 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff98814f220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9780097d0 tx=0x7ff97802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0c3b8f9c03e28c6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 1 -- 172.21.15.139:0/365170173 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978035020 con 0x7ff9880b3a80 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: dump: 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.227 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: _finish_auth 0 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: _check_auth_tickets 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 1 -- 172.21.15.139:0/365170173 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff978031890 con 0x7ff9880b3a80 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 5 monclient: authenticate success, global_id 4181 2024-07-26T23:34:44.228 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 4 set_mon_vals no callback set 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 1 -- 172.21.15.139:0/365170173 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978031c60 con 0x7ff9880b3a80 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff98d216640 10 monclient: dump: 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:44.229 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:44.230 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:44.230 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:44.230 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:44.230 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:44.230 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 10 monclient: shutdown 2024-07-26T23:34:44.230 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:44.230 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 1 -- 172.21.15.139:0/365170173 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 msgr2=0x7ff98814f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:44.231 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 1 --2- 172.21.15.139:0/365170173 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff98814f220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9780097d0 tx=0x7ff97802b930 comp rx=0 tx=0).stop 2024-07-26T23:34:44.231 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 1 -- 172.21.15.139:0/365170173 shutdown_connections 2024-07-26T23:34:44.231 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 1 --2- 172.21.15.139:0/365170173 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff98814f220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:44.231 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 1 -- 172.21.15.139:0/365170173 >> 172.21.15.139:0/365170173 conn(0x7ff9880b9c40 msgr2=0x7ff9880ba050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:44.231 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.225+0000 7ff99047c640 1 -- 172.21.15.139:0/365170173 shutdown_connections 2024-07-26T23:34:44.231 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.226+0000 7ff99047c640 1 -- 172.21.15.139:0/365170173 wait complete. 2024-07-26T23:34:44.234 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.226+0000 7ff99047c640 10 monclient: build_initial_monmap 2024-07-26T23:34:44.234 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.226+0000 7ff99047c640 10 monclient: monmap: 2024-07-26T23:34:44.234 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:44.234 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:44.234 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.226+0000 7ff99047c640 1 librados: starting msgr at 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.226+0000 7ff99047c640 1 librados: starting objecter 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 Processor -- start 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 -- start start 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 librados: setting wanted keys 2024-07-26T23:34:44.235 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 librados: calling monclient init 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient: init 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff98806e580 0x7ff98813d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): picked mon.noname-b con 0x7ff98806e580 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff988139b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): picked mon.noname-c con 0x7ff9880b3a80 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9881444b0 0x7ff98813a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): picked mon.noname-a con 0x7ff9881444b0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880b4960 con 0x7ff9881444b0 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:44.236 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880ab870 con 0x7ff98806e580 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff9880ae850 con 0x7ff9880b3a80 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.227+0000 7ff99047c640 10 monclient(hunting): authenticate will time out at 1317.263699s 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98da17640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff988139b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98ea19640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9881444b0 0x7ff98813a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98e218640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff98806e580 0x7ff98813d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98da17640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff988139b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60628/0 (socket says 172.21.15.139:60628) 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98da17640 1 -- 172.21.15.139:0/4137903634 learned_addr learned my addr 172.21.15.139:0/4137903634 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98da17640 10 monclient(hunting): get_auth_request con 0x7ff9880b3a80 auth_method 0 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98da17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98da17640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:44.237 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98da17640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98e218640 10 monclient(hunting): get_auth_request con 0x7ff98806e580 auth_method 0 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98e218640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98e218640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98e218640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98ea19640 10 monclient(hunting): get_auth_request con 0x7ff9881444b0 auth_method 0 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98ea19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98ea19640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.228+0000 7ff98ea19640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98da17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98da17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98da17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98e218640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:44.238 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98e218640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:44.239 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98e218640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:44.239 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:24.515 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:44.240 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98ea19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:44.240 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98ea19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:44.240 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.229+0000 7ff98ea19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 10 monclient(hunting): handle_auth_done global_id 4210 payload 824 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 1 -- 172.21.15.139:0/4137903634 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 msgr2=0x7ff988139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff988139b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 1 -- 172.21.15.139:0/4137903634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9881444b0 msgr2=0x7ff98813a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9881444b0 0x7ff98813a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 10 monclient: _finish_hunting 0 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 1 monclient: found mon.noname-b 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.232+0000 7ff98e218640 1 -- 172.21.15.139:0/4137903634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98813da30 con 0x7ff98806e580 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.234+0000 7ff98e218640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff98806e580 0x7ff98813d310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff98000c3c0 tx=0x7ff98000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:44.241 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.234+0000 7ff986ffd640 10 client.?.objecter ms_handle_connect 0x7ff98806e580 2024-07-26T23:34:44.242 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.234+0000 7ff986ffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:44.242 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:24.520 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff986ffd640 1 -- 172.21.15.139:0/4137903634 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9800074e0 con 0x7ff98806e580 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff986ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff986ffd640 10 monclient: dump: 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.244 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff986ffd640 10 monclient: _finish_auth 0 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff986ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 5 monclient: authenticate success, global_id 4210 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 10 monclient: _renew_subs 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:44.245 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9881d6540 con 0x7ff98806e580 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 10 monclient: _renew_subs 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9880ad450 con 0x7ff98806e580 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.236+0000 7ff99047c640 1 librados: init done 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.237+0000 7ff99047c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.237+0000 7ff99047c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.237+0000 7ff99047c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:44.246 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.237+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff98813d850 con 0x7ff98806e580 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.238+0000 7ff986ffd640 1 -- 172.21.15.139:0/4137903634 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9800047e0 con 0x7ff98806e580 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.238+0000 7ff986ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.238+0000 7ff986ffd640 1 -- 172.21.15.139:0/4137903634 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980004b90 con 0x7ff98806e580 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.238+0000 7ff986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.238+0000 7ff986ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.238+0000 7ff986ffd640 10 monclient: dump: 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:44.247 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:44.248 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:44.248 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:44.248 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:44.248 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.238+0000 7ff98ca15640 4 set_mon_vals no callback set 2024-07-26T23:34:44.269 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:24.539 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:44.272 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:24.543 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:44.412 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:24.689 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:44.417 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:24.694 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:44.676 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.674+0000 7ff986ffd640 1 -- 172.21.15.139:0/4137903634 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 3) ==== 99083+0+0 (secure 0 0 0) 0x7ff980011070 con 0x7ff98806e580 2024-07-26T23:34:44.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.675+0000 7ff986ffd640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff974078940 0x7ff97407ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:44.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.675+0000 7ff98da17640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff974078940 0x7ff97407ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:44.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.675+0000 7ff98da17640 10 monclient: get_auth_request con 0x7ff974078940 auth_method 0 2024-07-26T23:34:44.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.676+0000 7ff98da17640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff974078940 0x7ff97407ae00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff988141120 tx=0x7ff97c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:44.677 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.676+0000 7ff986ffd640 10 client.4210.objecter ms_handle_connect 0x7ff974078940 2024-07-26T23:34:44.681 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.680+0000 7ff986ffd640 1 -- 172.21.15.139:0/4137903634 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ff980096c60 con 0x7ff98806e580 2024-07-26T23:34:44.681 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.680+0000 7ff986ffd640 10 client.4210.objecter ms_dispatch 0x7ff988000b90 osd_map(10..10 src has 1..10) 2024-07-26T23:34:44.681 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.680+0000 7ff986ffd640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-07-26T23:34:44.681 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.680+0000 7ff986ffd640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-07-26T23:34:44.681 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.680+0000 7ff986ffd640 20 client.4210.objecter dump_active .. 0 homeless 2024-07-26T23:34:44.692 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.691+0000 7ff986ffd640 1 -- 172.21.15.139:0/4137903634 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7ff98813d850 con 0x7ff98806e580 2024-07-26T23:34:44.692 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.691+0000 7ff986ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:44.693 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.691+0000 7ff986ffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:44.886 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.884+0000 7ff99047c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:44.886 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.884+0000 7ff99047c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:44.886 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.884+0000 7ff99047c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:34:44.886 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.884+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9880b4250 con 0x7ff98806e580 2024-07-26T23:34:44.899 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.898+0000 7ff986ffd640 1 -- 172.21.15.139:0/4137903634 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ff9800612b0 con 0x7ff98806e580 2024-07-26T23:34:44.899 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.898+0000 7ff986ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:44.899 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.898+0000 7ff986ffd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:44.900 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:34:44.900 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":10,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:34:19.494450+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 10 librados: watch_flush enter 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 10 librados: watch_flush exit 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 20 client.4210.objecter shutdown clearing up homeless session... 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 10 client.4210.objecter successfully canceled tick 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff974078940 msgr2=0x7ff97407ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff974078940 0x7ff97407ae00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff988141120 tx=0x7ff97c009040 comp rx=0 tx=0).stop 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 10 monclient: shutdown 2024-07-26T23:34:44.902 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:44.903 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff98806e580 msgr2=0x7ff98813d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:44.903 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.900+0000 7ff99047c640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff98806e580 0x7ff98813d310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff98000c3c0 tx=0x7ff98000c880 comp rx=0 tx=0).stop 2024-07-26T23:34:44.903 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 shutdown_connections 2024-07-26T23:34:44.903 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff974078940 0x7ff97407ae00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:44.903 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff9880b3a80 0x7ff988139b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:44.904 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9881444b0 0x7ff98813a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:44.904 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 --2- 172.21.15.139:0/4137903634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff98806e580 0x7ff98813d310 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:44.904 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 >> 172.21.15.139:0/4137903634 conn(0x7ff9880b9c40 msgr2=0x7ff9880b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:44.904 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 shutdown_connections 2024-07-26T23:34:44.904 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.901+0000 7ff99047c640 1 -- 172.21.15.139:0/4137903634 wait complete. 2024-07-26T23:34:44.904 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:44.902+0000 7ff99047c640 1 librados: shutdown 2024-07-26T23:34:44.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-26T23:34:46.351 INFO:tasks.ceph.osd.5.smithi177.stderr:==00:00:00:26.648 35446== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:46.368 INFO:tasks.ceph.osd.6.smithi177.stderr:==00:00:00:26.660 35448== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:46.945 INFO:tasks.ceph.osd.7.smithi177.stderr:==00:00:00:27.236 35449== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:47.463 INFO:tasks.ceph.osd.4.smithi177.stderr:==00:00:00:27.760 35447== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:47.907 INFO:tasks.ceph.osd.1.smithi139.stderr:==00:00:00:28.187 35278== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:48.632 INFO:tasks.ceph.osd.0.smithi139.stderr:==00:00:00:28.916 35277== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:48.853 INFO:tasks.ceph.osd.3.smithi139.stderr:==00:00:00:29.123 35280== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:49.116 INFO:tasks.ceph.osd.2.smithi139.stderr:==00:00:00:29.393 35279== Warning: unimplemented fcntl command: 1038 2024-07-26T23:34:51.723 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:34:51.834 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.833+0000 7fb55a0ce640 10 monclient: get_monmap_and_config 2024-07-26T23:34:51.834 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.833+0000 7fb55a0ce640 10 monclient: build_initial_monmap 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.833+0000 7fb55a0ce640 10 monclient: monmap: 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:34:51.834234+0000 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:34:51.834234+0000 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:51.835 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:51.836 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:51.836 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:51.836 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:51.836 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:51.836 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:51.838 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.833+0000 7fb55a0ce640 1 Processor -- start 2024-07-26T23:34:51.838 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 1 -- start start 2024-07-26T23:34:51.838 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 10 monclient: init 2024-07-26T23:34:51.839 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:51.839 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-07-26T23:34:51.839 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5540b62c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:51.839 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb554143970 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:51.839 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb55413a500 0x7fb5540b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:51.839 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 10 monclient(hunting): picked mon.noname-a con 0x7fb55413a500 addr v2:172.21.15.139:3300/0 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5540acc10 con 0x7fb554143970 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.834+0000 7fb55a0ce640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb554144490 con 0x7fb5540b62c0 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.835+0000 7fb55a0ce640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.835+0000 7fb55a0ce640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fb5540b4960 con 0x7fb55413a500 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.835+0000 7fb55a0ce640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.835+0000 7fb55a0ce640 10 monclient(hunting): authenticate will time out at 1324.871021s 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.835+0000 7fb553fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb55413a500 0x7fb5540b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.835+0000 7fb5537fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb5540b62c0 0x7fb5540b4250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:51978/0 (socket says 172.21.15.139:51978) 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.835+0000 7fb5537fe640 1 -- 172.21.15.139:0/4254810630 learned_addr learned my addr 172.21.15.139:0/4254810630 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:34:51.840 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.836+0000 7fb553fff640 10 monclient(hunting): get_auth_request con 0x7fb55413a500 auth_method 0 2024-07-26T23:34:51.841 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.836+0000 7fb553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:51.841 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.836+0000 7fb553fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:51.841 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.836+0000 7fb553fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:51.841 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.837+0000 7fb553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:51.841 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.837+0000 7fb553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:51.841 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.837+0000 7fb553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:51.842 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-07-26T23:34:51.842 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 1 -- 172.21.15.139:0/4254810630 >> v1:172.21.15.177:6789/0 conn(0x7fb5540b62c0 legacy=0x7fb5540b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:51.842 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 1 -- 172.21.15.139:0/4254810630 >> v1:172.21.15.139:6789/0 conn(0x7fb554143970 legacy=0x7fb554139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:51.842 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:51.843 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 1 monclient: found mon.noname-a 2024-07-26T23:34:51.844 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:51.844 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:34:51.844 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.839+0000 7fb553fff640 1 -- 172.21.15.139:0/4254810630 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb554144660 con 0x7fb55413a500 2024-07-26T23:34:51.844 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.842+0000 7fb553fff640 1 --2- 172.21.15.139:0/4254810630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb5540b66a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009d90 tx=0x7fb54c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a9d9e5c39ab9de server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:51.847 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb5527fc640 1 -- 172.21.15.139:0/4254810630 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c039070 con 0x7fb55413a500 2024-07-26T23:34:51.847 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:51.847 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb5527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:34:51.847 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb5527fc640 10 monclient: dump: 2024-07-26T23:34:51.847 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb5527fc640 10 monclient: _finish_auth 0 2024-07-26T23:34:51.848 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb5527fc640 10 monclient: _check_auth_tickets 2024-07-26T23:34:51.849 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb5527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:51.849 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb55a0ce640 5 monclient: authenticate success, global_id 4221 2024-07-26T23:34:51.849 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.846+0000 7fb55a0ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:34:51.849 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb5527fc640 1 -- 172.21.15.139:0/4254810630 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb54c02cbd0 con 0x7fb55413a500 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb5527fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb5527fc640 1 -- 172.21.15.139:0/4254810630 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c034440 con 0x7fb55413a500 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb5527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb5527fc640 10 monclient: dump: 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:51.850 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb55a0ce640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.847+0000 7fb55a0ce640 4 set_mon_vals no callback set 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 10 monclient: shutdown 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 1 -- 172.21.15.139:0/4254810630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 msgr2=0x7fb5540b66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/4254810630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb5540b66a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009d90 tx=0x7fb54c02be20 comp rx=0 tx=0).stop 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 1 -- 172.21.15.139:0/4254810630 shutdown_connections 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/4254810630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb5540b66a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 1 -- 172.21.15.139:0/4254810630 >> 172.21.15.139:0/4254810630 conn(0x7fb55406e710 msgr2=0x7fb55402d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 1 -- 172.21.15.139:0/4254810630 shutdown_connections 2024-07-26T23:34:51.851 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.848+0000 7fb55a0ce640 1 -- 172.21.15.139:0/4254810630 wait complete. 2024-07-26T23:34:51.853 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 10 monclient: build_initial_monmap 2024-07-26T23:34:51.853 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 10 monclient: monmap: 2024-07-26T23:34:51.854 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:51.855 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 1 librados: starting msgr at 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 1 librados: starting objecter 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 1 Processor -- start 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 1 -- start start 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 1 librados: setting wanted keys 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 1 librados: calling monclient init 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.849+0000 7fb55a0ce640 10 monclient: init 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb5540b62c0 0x7fb55413e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): picked mon.noname-c con 0x7fb5540b62c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:51.856 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb55413eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): picked mon.noname-a con 0x7fb55413a500 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb554143970 0x7fb55413f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): picked mon.noname-b con 0x7fb554143970 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb55414f620 con 0x7fb55413a500 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb55414f4a0 con 0x7fb554143970 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb55414f7a0 con 0x7fb5540b62c0 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.850+0000 7fb55a0ce640 10 monclient(hunting): authenticate will time out at 1324.886680s 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb55413eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:51.857 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb5537fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb5540b62c0 0x7fb55413e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb55413eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36482/0 (socket says 172.21.15.139:36482) 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 1 -- 172.21.15.139:0/2013439918 learned_addr learned my addr 172.21.15.139:0/2013439918 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 10 monclient(hunting): get_auth_request con 0x7fb55413a500 auth_method 0 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb553fff640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb554143970 0x7fb55413f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb5537fe640 10 monclient(hunting): get_auth_request con 0x7fb5540b62c0 auth_method 0 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb5537fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb5537fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:51.858 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb553fff640 10 monclient(hunting): get_auth_request con 0x7fb554143970 auth_method 0 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb553fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.851+0000 7fb553fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.852+0000 7fb5537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.852+0000 7fb5537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:51.859 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.852+0000 7fb5537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.852+0000 7fb553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.852+0000 7fb553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.852+0000 7fb553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 1 -- 172.21.15.139:0/2013439918 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb5540b62c0 msgr2=0x7fb55413e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb5540b62c0 0x7fb55413e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 1 -- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb554143970 msgr2=0x7fb55413f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:51.866 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb554143970 0x7fb55413f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 1 monclient: found mon.noname-a 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.854+0000 7fb552ffd640 1 -- 172.21.15.139:0/2013439918 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5541406f0 con 0x7fb55413a500 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.857+0000 7fb53bfff640 10 client.?.objecter ms_handle_connect 0x7fb55413a500 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.857+0000 7fb53bfff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.857+0000 7fb552ffd640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb55413eb60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c400 tx=0x7fb54800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 1 -- 172.21.15.139:0/2013439918 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548017070 con 0x7fb55413a500 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 10 monclient: dump: 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:51.867 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 10 monclient: _finish_auth 0 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 10 monclient: _check_auth_tickets 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 1 -- 172.21.15.139:0/2013439918 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5480052d0 con 0x7fb55413a500 2024-07-26T23:34:51.868 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb53bfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb551ffb640 4 set_mon_vals no callback set 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 5 monclient: authenticate success, global_id 4224 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 10 monclient: _renew_subs 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5541409d0 con 0x7fb55413a500 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 10 monclient: _renew_subs 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb55413f5e0 con 0x7fb55413a500 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.861+0000 7fb55a0ce640 1 librados: init done 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb53bfff640 1 -- 172.21.15.139:0/2013439918 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548017070 con 0x7fb55413a500 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb53bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:34:51.869 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb53bfff640 10 monclient: dump: 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb55a0ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb55a0ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:51.870 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb55a0ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:51.871 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.862+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5540b66a0 con 0x7fb55413a500 2024-07-26T23:34:51.890 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.888+0000 7fb53bfff640 1 -- 172.21.15.139:0/2013439918 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 3) ==== 99083+0+0 (secure 0 0 0) 0x7fb54801a070 con 0x7fb55413a500 2024-07-26T23:34:51.890 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.889+0000 7fb53bfff640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb540078b10 0x7fb54007afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:51.890 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.889+0000 7fb5537fe640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb540078b10 0x7fb54007afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:51.890 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.889+0000 7fb5537fe640 10 monclient: get_auth_request con 0x7fb540078b10 auth_method 0 2024-07-26T23:34:51.891 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.890+0000 7fb5537fe640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb540078b10 0x7fb54007afd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb544000f60 tx=0x7fb544009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:51.891 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.890+0000 7fb53bfff640 10 client.4224.objecter ms_handle_connect 0x7fb540078b10 2024-07-26T23:34:51.895 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.893+0000 7fb53bfff640 1 -- 172.21.15.139:0/2013439918 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fb548096380 con 0x7fb55413a500 2024-07-26T23:34:51.895 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.893+0000 7fb53bfff640 10 client.4224.objecter ms_dispatch 0x7fb554000b90 osd_map(10..10 src has 1..10) 2024-07-26T23:34:51.895 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.893+0000 7fb53bfff640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2024-07-26T23:34:51.895 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.893+0000 7fb53bfff640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2024-07-26T23:34:51.895 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:51.893+0000 7fb53bfff640 20 client.4224.objecter dump_active .. 0 homeless 2024-07-26T23:34:52.326 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.324+0000 7fb53bfff640 1 -- 172.21.15.139:0/2013439918 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fb548060a60 con 0x7fb55413a500 2024-07-26T23:34:52.326 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.324+0000 7fb53bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:52.326 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.324+0000 7fb53bfff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:34:52.646 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.644+0000 7fb55a0ce640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:52.646 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.644+0000 7fb55a0ce640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:52.646 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.644+0000 7fb55a0ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:34:52.646 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.644+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb554076800 con 0x7fb55413a500 2024-07-26T23:34:52.665 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.663+0000 7fb53bfff640 1 -- 172.21.15.139:0/2013439918 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fb5480692f0 con 0x7fb55413a500 2024-07-26T23:34:52.665 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.663+0000 7fb53bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:34:52.665 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.663+0000 7fb53bfff640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:34:52.665 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:34:52.666 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":10,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:34:19.494450+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:34:52.668 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 10 librados: watch_flush enter 2024-07-26T23:34:52.668 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 10 librados: watch_flush exit 2024-07-26T23:34:52.668 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 20 client.4224.objecter shutdown clearing up homeless session... 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 10 client.4224.objecter successfully canceled tick 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb540078b10 msgr2=0x7fb54007afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb540078b10 0x7fb54007afd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb544000f60 tx=0x7fb544009040 comp rx=0 tx=0).stop 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 10 monclient: shutdown 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 msgr2=0x7fb55413eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb55413eb60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c400 tx=0x7fb54800c8c0 comp rx=0 tx=0).stop 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 shutdown_connections 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb540078b10 0x7fb54007afd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb554143970 0x7fb55413f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:52.669 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb55413a500 0x7fb55413eb60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:52.670 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 --2- 172.21.15.139:0/2013439918 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb5540b62c0 0x7fb55413e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:52.670 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.666+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 >> 172.21.15.139:0/2013439918 conn(0x7fb55406e710 msgr2=0x7fb5540b88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:52.670 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.667+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 shutdown_connections 2024-07-26T23:34:52.670 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.667+0000 7fb55a0ce640 1 -- 172.21.15.139:0/2013439918 wait complete. 2024-07-26T23:34:52.671 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:52.667+0000 7fb55a0ce640 1 librados: shutdown 2024-07-26T23:34:52.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-26T23:34:55.251 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-07-26T23:34:55.244+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-07-26T23:34:55.391 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-07-26T23:34:55.357+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-07-26T23:34:55.663 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-07-26T23:34:55.656+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-07-26T23:34:55.847 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:34:55.841+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-07-26T23:34:57.141 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:34:57.129+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-07-26T23:34:57.626 INFO:tasks.ceph.osd.3.smithi139.stderr:2024-07-26T23:34:57.616+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-07-26T23:34:57.936 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:34:57.926+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-07-26T23:34:58.460 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-07-26T23:34:58.456+0000 bc99640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T23:34:58.471 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-07-26T23:34:58.466+0000 bc99640 -1 osd.6 0 waiting for initial osdmap 2024-07-26T23:34:58.537 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:34:58.530+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-07-26T23:34:58.875 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-07-26T23:34:58.859+0000 10261640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:34:58.908 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-07-26T23:34:58.893+0000 10261640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:34:59.488 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:34:59.586 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.585+0000 7f95989f8640 10 monclient: get_monmap_and_config 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.585+0000 7f95989f8640 10 monclient: build_initial_monmap 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.585+0000 7f95989f8640 10 monclient: monmap: 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:34:59.586382+0000 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:34:59.586382+0000 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:34:59.587 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:34:59.588 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:59.590 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 1 Processor -- start 2024-07-26T23:34:59.590 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 1 -- start start 2024-07-26T23:34:59.590 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient: init 2024-07-26T23:34:59.590 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:59.590 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-07-26T23:34:59.590 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95940b62c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9594143970 addr v1:172.21.15.139:6789/0 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f95940b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient(hunting): picked mon.noname-b con 0x7f959413a500 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95940acc10 con 0x7f9594143970 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9594144490 con 0x7f95940b62c0 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f95940b4960 con 0x7f959413a500 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.586+0000 7f95989f8640 10 monclient(hunting): authenticate will time out at 1332.622836s 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.587+0000 7f9592d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f95940b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.587+0000 7f958bfff640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f9594143970 0x7f9594139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:44892/0 (socket says 172.21.15.139:44892) 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.587+0000 7f958bfff640 1 -- 172.21.15.139:0/2573073924 learned_addr learned my addr 172.21.15.139:0/2573073924 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.587+0000 7f9592d9d640 10 monclient(hunting): get_auth_request con 0x7f959413a500 auth_method 0 2024-07-26T23:34:59.591 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.587+0000 7f9592d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:59.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.587+0000 7f9592d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:59.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.587+0000 7f9592d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:59.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.588+0000 7f9592d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:59.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.588+0000 7f9592d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:59.592 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.588+0000 7f9592d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9591d9b640 1 -- 172.21.15.139:0/2573073924 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908427701 0 0) 0x7f95940acc10 con 0x7f9594143970 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 1 -- 172.21.15.139:0/2573073924 >> v1:172.21.15.177:6789/0 conn(0x7f95940b62c0 legacy=0x7f95940b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 1 -- 172.21.15.139:0/2573073924 >> v1:172.21.15.139:6789/0 conn(0x7f9594143970 legacy=0x7f9594139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 10 monclient: _finish_hunting 0 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 1 monclient: found mon.noname-b 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9592d9d640 1 -- 172.21.15.139:0/2573073924 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9594144660 con 0x7f959413a500 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.590+0000 7f9591d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-07-26T23:34:59.593 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.591+0000 7f9592d9d640 1 --2- 172.21.15.139:0/2573073924 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f95940b66a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f958c009dc0 tx=0x7f958c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2971853811fcd7ff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:59.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 1 -- 172.21.15.139:0/2573073924 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c039070 con 0x7f959413a500 2024-07-26T23:34:59.597 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:59.598 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:34:59.598 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: dump: 2024-07-26T23:34:59.598 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:59.598 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:59.598 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.598 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: _finish_auth 0 2024-07-26T23:34:59.599 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 1 -- 172.21.15.139:0/2573073924 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958c02cbb0 con 0x7f959413a500 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 1 -- 172.21.15.139:0/2573073924 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c034440 con 0x7f959413a500 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f9591d9b640 10 monclient: dump: 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:59.600 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f95989f8640 5 monclient: authenticate success, global_id 4211 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f95989f8640 10 monclient: get_monmap_and_config success 2024-07-26T23:34:59.601 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.596+0000 7f95989f8640 4 set_mon_vals no callback set 2024-07-26T23:34:59.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 10 monclient: shutdown 2024-07-26T23:34:59.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:34:59.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 1 -- 172.21.15.139:0/2573073924 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 msgr2=0x7f95940b66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:59.605 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 1 --2- 172.21.15.139:0/2573073924 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f95940b66a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f958c009dc0 tx=0x7f958c02be00 comp rx=0 tx=0).stop 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 1 -- 172.21.15.139:0/2573073924 shutdown_connections 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 1 --2- 172.21.15.139:0/2573073924 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f95940b66a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 1 -- 172.21.15.139:0/2573073924 >> 172.21.15.139:0/2573073924 conn(0x7f959406e710 msgr2=0x7f959402d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 1 -- 172.21.15.139:0/2573073924 shutdown_connections 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.597+0000 7f95989f8640 1 -- 172.21.15.139:0/2573073924 wait complete. 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 10 monclient: build_initial_monmap 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 10 monclient: monmap: 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:34:59.606 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 1 librados: starting msgr at 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 1 librados: starting objecter 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 1 Processor -- start 2024-07-26T23:34:59.607 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 1 -- start start 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 1 librados: setting wanted keys 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 1 librados: calling monclient init 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.598+0000 7f95989f8640 10 monclient: init 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient: _reopen_session rank -1 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95940b62c0 0x7f959413b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): picked mon.noname-a con 0x7f95940b62c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f959414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): picked mon.noname-c con 0x7f959413a500 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9594143970 0x7f959414eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): picked mon.noname-b con 0x7f9594143970 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:59.608 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f959414f620 con 0x7f95940b62c0 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f959414f4a0 con 0x7f9594143970 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f959414f7a0 con 0x7f959413a500 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): _renew_subs 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.599+0000 7f95989f8640 10 monclient(hunting): authenticate will time out at 1332.635488s 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f958bfff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f959414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f959259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95940b62c0 0x7f959413b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f9592d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9594143970 0x7f959414eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f959259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95940b62c0 0x7f959413b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36526/0 (socket says 172.21.15.139:36526) 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f959259c640 1 -- 172.21.15.139:0/1808238158 learned_addr learned my addr 172.21.15.139:0/1808238158 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f959259c640 10 monclient(hunting): get_auth_request con 0x7f95940b62c0 auth_method 0 2024-07-26T23:34:59.609 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f959259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f959259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f959259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f9592d9d640 10 monclient(hunting): get_auth_request con 0x7f9594143970 auth_method 0 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f9592d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f9592d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f9592d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f958bfff640 10 monclient(hunting): get_auth_request con 0x7f959413a500 auth_method 0 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f958bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f958bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.600+0000 7f958bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f958bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f958bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:59.610 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f958bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f959259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f959259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f959259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f9592d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f9592d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.601+0000 7f9592d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 1 -- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9594143970 msgr2=0x7f959414eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9594143970 0x7f959414eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 1 -- 172.21.15.139:0/1808238158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95940b62c0 msgr2=0x7f959413b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95940b62c0 0x7f959413b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 1 monclient: found mon.noname-c 2024-07-26T23:34:59.611 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:34:59.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:34:59.612 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.603+0000 7f958bfff640 1 -- 172.21.15.139:0/1808238158 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959413db10 con 0x7f959413a500 2024-07-26T23:34:59.612 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-07-26T23:34:59.607+0000 bc99640 -1 osd.7 0 waiting for initial osdmap 2024-07-26T23:34:59.613 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.606+0000 7f958bfff640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f959414e5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c3c0 tx=0x7f957c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:59.613 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.606+0000 7f958affd640 10 client.?.objecter ms_handle_connect 0x7f959413a500 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.606+0000 7f958affd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 1 -- 172.21.15.139:0/1808238158 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c0074e0 con 0x7f959413a500 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: dump: 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.614 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: _finish_auth 0 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: _check_auth_tickets 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 1 -- 172.21.15.139:0/1808238158 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f957c0047e0 con 0x7f959413a500 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:34:59.615 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 1 -- 172.21.15.139:0/1808238158 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c004bd0 con 0x7f959413a500 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f958affd640 10 monclient: dump: 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:34:59.616 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 5 monclient: authenticate success, global_id 4214 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 10 monclient: _renew_subs 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959413c790 con 0x7f959413a500 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f959159a640 4 set_mon_vals no callback set 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 10 monclient: _renew_subs 2024-07-26T23:34:59.617 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:59.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f959413bbf0 con 0x7f959413a500 2024-07-26T23:34:59.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.609+0000 7f95989f8640 1 librados: init done 2024-07-26T23:34:59.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.610+0000 7f95989f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:59.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.610+0000 7f95989f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:59.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.610+0000 7f95989f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:34:59.618 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.610+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95940b66a0 con 0x7f959413a500 2024-07-26T23:34:59.632 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.631+0000 7f958affd640 1 -- 172.21.15.139:0/1808238158 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 3) ==== 99083+0+0 (secure 0 0 0) 0x7f957c011070 con 0x7f959413a500 2024-07-26T23:34:59.632 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.631+0000 7f958affd640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9574078990 0x7f957407ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:34:59.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.631+0000 7f959259c640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9574078990 0x7f957407ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:34:59.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.631+0000 7f959259c640 10 monclient: get_auth_request con 0x7f9574078990 auth_method 0 2024-07-26T23:34:59.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.632+0000 7f958affd640 10 client.4214.objecter ms_handle_connect 0x7f9574078990 2024-07-26T23:34:59.633 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.632+0000 7f959259c640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9574078990 0x7f957407ae50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9584000f60 tx=0x7f9584009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:34:59.878 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 1 -- 172.21.15.139:0/1808238158 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 2575+0+0 (secure 0 0 0) 0x7f957c05d720 con 0x7f959413a500 2024-07-26T23:34:59.878 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 10 client.4214.objecter ms_dispatch 0x7f9594000b90 osd_map(13..13 src has 1..13) 2024-07-26T23:34:59.878 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 3 client.4214.objecter handle_osd_map got epochs [13,13] > 0 2024-07-26T23:34:59.878 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 3 client.4214.objecter handle_osd_map decoding full epoch 13 2024-07-26T23:34:59.878 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 20 client.4214.objecter dump_active .. 0 homeless 2024-07-26T23:34:59.878 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 1 -- 172.21.15.139:0/1808238158 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f957c061490 con 0x7f959413a500 2024-07-26T23:34:59.879 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:34:59.879 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:34:59.876+0000 7f958affd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:00.080 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.078+0000 7f95989f8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:00.080 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.078+0000 7f95989f8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:00.080 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.078+0000 7f95989f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:00.080 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.078+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f959413d930 con 0x7f959413a500 2024-07-26T23:35:00.105 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.103+0000 7f958affd640 1 -- 172.21.15.139:0/1808238158 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+8041 (secure 0 0 0) 0x7f959413d930 con 0x7f959413a500 2024-07-26T23:35:00.105 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.104+0000 7f958affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:00.105 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.104+0000 7f958affd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:00.106 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:35:00.106 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":13,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:34:59.163151+0000","last_up_change":"2024-07-26T23:34:59.163151+0000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6803","nonce":228413488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6805","nonce":228413488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6809","nonce":228413488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6807","nonce":228413488}]},"public_addr":"172.21.15.177:6803/228413488","cluster_addr":"172.21.15.177:6805/228413488","heartbeat_back_addr":"172.21.15.177:6809/228413488","heartbeat_front_addr":"172.21.15.177:6807/228413488","state":["exists","up"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6811","nonce":1761005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6813","nonce":1761005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6817","nonce":1761005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6815","nonce":1761005298}]},"public_addr":"172.21.15.177:6811/1761005298","cluster_addr":"172.21.15.177:6813/1761005298","heartbeat_back_addr":"172.21.15.177:6817/1761005298","heartbeat_front_addr":"172.21.15.177:6815/1761005298","state":["exists","up"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:35:00.108 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 10 librados: watch_flush enter 2024-07-26T23:35:00.108 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 10 librados: watch_flush exit 2024-07-26T23:35:00.108 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 20 client.4214.objecter shutdown clearing up homeless session... 2024-07-26T23:35:00.108 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 10 client.4214.objecter successfully canceled tick 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9574078990 msgr2=0x7f957407ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9574078990 0x7f957407ae50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9584000f60 tx=0x7f9584009040 comp rx=0 tx=0).stop 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 10 monclient: shutdown 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 msgr2=0x7f959414e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f959414e5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c3c0 tx=0x7f957c00c880 comp rx=0 tx=0).stop 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.106+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 shutdown_connections 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9574078990 0x7f957407ae50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f959413a500 0x7f959414e5c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9594143970 0x7f959414eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 --2- 172.21.15.139:0/1808238158 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f95940b62c0 0x7f959413b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 >> 172.21.15.139:0/1808238158 conn(0x7f959406e710 msgr2=0x7f95941419c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:00.109 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 shutdown_connections 2024-07-26T23:35:00.110 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 -- 172.21.15.139:0/1808238158 wait complete. 2024-07-26T23:35:00.110 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:00.107+0000 7f95989f8640 1 librados: shutdown 2024-07-26T23:35:00.119 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-07-26T23:35:00.278 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-07-26T23:35:00.262+0000 10261640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:35:06.591 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:06.552+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 3 2024-07-26T23:35:06.927 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:35:06.981 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:06.592+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-07-26T23:35:07.030 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.029+0000 7f5419cbc640 10 monclient: get_monmap_and_config 2024-07-26T23:35:07.034 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.029+0000 7f5419cbc640 10 monclient: build_initial_monmap 2024-07-26T23:35:07.034 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.029+0000 7f5419cbc640 10 monclient: monmap: 2024-07-26T23:35:07.034 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:35:07.034 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:07.035 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:35:07.030129+0000 2024-07-26T23:35:07.035 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:35:07.030129+0000 2024-07-26T23:35:07.035 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:07.035 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:07.035 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:07.035 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:07.035 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:07.036 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:07.036 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:07.036 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:07.040 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.029+0000 7f5419cbc640 1 Processor -- start 2024-07-26T23:35:07.040 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.029+0000 7f5419cbc640 1 -- start start 2024-07-26T23:35:07.040 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.029+0000 7f5419cbc640 10 monclient: init 2024-07-26T23:35:07.040 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:07.040 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-07-26T23:35:07.041 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54140b62c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:07.041 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5414143970 0x7f54140b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:07.041 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient(hunting): picked mon.noname-c con 0x7f5414143970 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:07.041 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f5414144490 0x7f5414139df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:07.041 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient(hunting): picked mon.noname-a con 0x7f5414144490 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:07.041 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54140acc10 con 0x7f54140b62c0 2024-07-26T23:35:07.041 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f54140b4960 con 0x7f5414144490 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f541413a330 con 0x7f5414143970 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.030+0000 7f5419cbc640 10 monclient(hunting): authenticate will time out at 1340.066882s 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.031+0000 7f5412ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5414143970 0x7f54140b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.031+0000 7f5413fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f5414144490 0x7f5414139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5414143970 0x7f54140b66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:52630/0 (socket says 172.21.15.139:52630) 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 1 -- 172.21.15.139:0/4090356059 learned_addr learned my addr 172.21.15.139:0/4090356059 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 10 monclient(hunting): get_auth_request con 0x7f5414143970 auth_method 0 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:07.042 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5413fff640 10 monclient(hunting): get_auth_request con 0x7f5414144490 auth_method 0 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5413fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5413fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5413fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.032+0000 7f5412ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.033+0000 7f5413fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.033+0000 7f5413fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.033+0000 7f5413fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2024-07-26T23:35:07.043 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 1 -- 172.21.15.139:0/4090356059 >> v2:172.21.15.177:3300/0 conn(0x7f5414143970 msgr2=0x7f54140b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 1 --2- 172.21.15.139:0/4090356059 >> v2:172.21.15.177:3300/0 conn(0x7f5414143970 0x7f54140b66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 1 -- 172.21.15.139:0/4090356059 >> v1:172.21.15.139:6789/0 conn(0x7f54140b62c0 legacy=0x7f54140b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 1 monclient: found mon.noname-a 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.035+0000 7f5413fff640 1 -- 172.21.15.139:0/4090356059 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541413a4b0 con 0x7f5414144490 2024-07-26T23:35:07.044 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.037+0000 7f5413fff640 1 --2- 172.21.15.139:0/4090356059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 0x7f5414139df0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f54140b7410 tx=0x7f540c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffbd5527cf74095 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:07.048 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 1 -- 172.21.15.139:0/4090356059 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c039070 con 0x7f5414144490 2024-07-26T23:35:07.048 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:07.048 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:07.048 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: dump: 2024-07-26T23:35:07.048 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:07.049 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f5419cbc640 5 monclient: authenticate success, global_id 4260 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f5419cbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 1 -- 172.21.15.139:0/4090356059 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f540c02cb40 con 0x7f5414144490 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f5419cbc640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f5419cbc640 4 set_mon_vals no callback set 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 1 -- 172.21.15.139:0/4090356059 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c034440 con 0x7f5414144490 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.045+0000 7f54127fc640 10 monclient: dump: 2024-07-26T23:35:07.050 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 10 monclient: shutdown 2024-07-26T23:35:07.051 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 -- 172.21.15.139:0/4090356059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 msgr2=0x7f5414139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 --2- 172.21.15.139:0/4090356059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 0x7f5414139df0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f54140b7410 tx=0x7f540c02bd90 comp rx=0 tx=0).stop 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 -- 172.21.15.139:0/4090356059 shutdown_connections 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 --2- 172.21.15.139:0/4090356059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 0x7f5414139df0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 --2- 172.21.15.139:0/4090356059 >> v2:172.21.15.177:3300/0 conn(0x7f5414143970 0x7f54140b66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 -- 172.21.15.139:0/4090356059 >> 172.21.15.139:0/4090356059 conn(0x7f541406e710 msgr2=0x7f541402d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 -- 172.21.15.139:0/4090356059 shutdown_connections 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.046+0000 7f5419cbc640 1 -- 172.21.15.139:0/4090356059 wait complete. 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.047+0000 7f5419cbc640 10 monclient: build_initial_monmap 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.047+0000 7f5419cbc640 10 monclient: monmap: 2024-07-26T23:35:07.052 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:07.053 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:07.055 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.047+0000 7f5419cbc640 1 librados: starting msgr at 2024-07-26T23:35:07.055 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.047+0000 7f5419cbc640 1 librados: starting objecter 2024-07-26T23:35:07.056 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 1 Processor -- start 2024-07-26T23:35:07.056 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 1 -- start start 2024-07-26T23:35:07.056 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 1 librados: setting wanted keys 2024-07-26T23:35:07.056 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 1 librados: calling monclient init 2024-07-26T23:35:07.056 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 10 monclient: init 2024-07-26T23:35:07.056 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:07.056 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54140b62c0 0x7f541413f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 10 monclient(hunting): picked mon.noname-b con 0x7f54140b62c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5414143970 0x7f541413b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 10 monclient(hunting): picked mon.noname-c con 0x7f5414143970 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.048+0000 7f5419cbc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 0x7f541413bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 10 monclient(hunting): picked mon.noname-a con 0x7f5414144490 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f541414e490 con 0x7f5414144490 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f541414e610 con 0x7f54140b62c0 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:07.057 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f541414e790 con 0x7f5414143970 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.049+0000 7f5419cbc640 10 monclient(hunting): authenticate will time out at 1340.085040s 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f54137fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54140b62c0 0x7f541413f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f54137fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54140b62c0 0x7f541413f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:52634/0 (socket says 172.21.15.139:52634) 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f54137fe640 1 -- 172.21.15.139:0/1687573127 learned_addr learned my addr 172.21.15.139:0/1687573127 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f54137fe640 10 monclient(hunting): get_auth_request con 0x7f54140b62c0 auth_method 0 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f54137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f54137fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f54137fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f5413fff640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 0x7f541413bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f5413fff640 10 monclient(hunting): get_auth_request con 0x7f5414144490 auth_method 0 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f5413fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:07.058 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f5413fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:07.059 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.050+0000 7f5413fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:07.059 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.052+0000 7f5413fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:07.059 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.052+0000 7f5413fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:07.059 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.052+0000 7f5413fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:07.059 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.055+0000 7f54137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:07.059 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.055+0000 7f54137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:07.060 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.055+0000 7f54137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:07.060 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2024-07-26T23:35:07.060 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 1 -- 172.21.15.139:0/1687573127 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5414143970 msgr2=0x7f541413b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:07.060 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5414143970 0x7f541413b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.060 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 1 -- 172.21.15.139:0/1687573127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 msgr2=0x7f541413bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:07.060 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 0x7f541413bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.060 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:07.061 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 1 monclient: found mon.noname-b 2024-07-26T23:35:07.061 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:07.061 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:07.061 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.059+0000 7f54137fe640 1 -- 172.21.15.139:0/1687573127 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54141d8bd0 con 0x7f54140b62c0 2024-07-26T23:35:07.062 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.060+0000 7f54137fe640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54140b62c0 0x7f541413f490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f540400b3f0 tx=0x7f540400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:07.062 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.060+0000 7f53fbfff640 10 client.?.objecter ms_handle_connect 0x7f54140b62c0 2024-07-26T23:35:07.062 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.060+0000 7f53fbfff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:07.063 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:06.981+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f53fbfff640 1 -- 172.21.15.139:0/1687573127 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404015020 con 0x7f54140b62c0 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f53fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f53fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f53fbfff640 10 monclient: dump: 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.065 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.066 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:07.066 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:07.066 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:07.066 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:07.066 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:07.066 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f53fbfff640 10 monclient: _finish_auth 0 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f53fbfff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f53fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f5419cbc640 5 monclient: authenticate success, global_id 4258 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f5419cbc640 10 monclient: _renew_subs 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f5419cbc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541413c4c0 con 0x7f54140b62c0 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f5419cbc640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.062+0000 7f5419cbc640 10 monclient: _renew_subs 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.063+0000 7f5419cbc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.063+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54141d9570 con 0x7f54140b62c0 2024-07-26T23:35:07.067 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.063+0000 7f5419cbc640 1 librados: init done 2024-07-26T23:35:07.069 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.064+0000 7f5419cbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:07.069 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.064+0000 7f5419cbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:07.070 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.064+0000 7f5419cbc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:07.070 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.064+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54141d8f70 con 0x7f54140b62c0 2024-07-26T23:35:07.070 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.065+0000 7f53fbfff640 1 -- 172.21.15.139:0/1687573127 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5404004020 con 0x7f54140b62c0 2024-07-26T23:35:07.070 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.065+0000 7f53fbfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:07.070 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.065+0000 7f53fbfff640 1 -- 172.21.15.139:0/1687573127 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404009200 con 0x7f54140b62c0 2024-07-26T23:35:07.070 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.065+0000 7f53fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:07.070 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.065+0000 7f53fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:07.071 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.065+0000 7f53fbfff640 10 monclient: dump: 2024-07-26T23:35:07.071 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:07.071 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:07.071 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.071 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:07.071 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:07.071 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:07.072 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:07.072 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:07.072 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:07.072 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:07.072 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.065+0000 7f5411ffb640 4 set_mon_vals no callback set 2024-07-26T23:35:07.221 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.063+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-07-26T23:35:07.245 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.220+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-07-26T23:35:07.279 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.244+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-07-26T23:35:07.288 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.257+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-07-26T23:35:07.342 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.340+0000 7f53fbfff640 1 -- 172.21.15.139:0/1687573127 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f5404018020 con 0x7f54140b62c0 2024-07-26T23:35:07.342 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.341+0000 7f53fbfff640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f54000789a0 0x7f540007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:07.342 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.341+0000 7f5412ffd640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f54000789a0 0x7f540007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:07.343 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.341+0000 7f5412ffd640 10 monclient: get_auth_request con 0x7f54000789a0 auth_method 0 2024-07-26T23:35:07.343 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.342+0000 7f53fbfff640 10 client.4258.objecter ms_handle_connect 0x7f54000789a0 2024-07-26T23:35:07.343 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.342+0000 7f5412ffd640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f54000789a0 0x7f540007ae60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f54080094a0 tx=0x7f54080093f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:07.347 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.345+0000 7f53fbfff640 1 -- 172.21.15.139:0/1687573127 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3457+0+0 (secure 0 0 0) 0x7f5404004430 con 0x7f54140b62c0 2024-07-26T23:35:07.347 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.345+0000 7f53fbfff640 10 client.4258.objecter ms_dispatch 0x7f5414000b90 osd_map(17..17 src has 1..17) 2024-07-26T23:35:07.347 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.345+0000 7f53fbfff640 3 client.4258.objecter handle_osd_map got epochs [17,17] > 0 2024-07-26T23:35:07.347 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.345+0000 7f53fbfff640 3 client.4258.objecter handle_osd_map decoding full epoch 17 2024-07-26T23:35:07.347 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.345+0000 7f53fbfff640 20 client.4258.objecter dump_active .. 0 homeless 2024-07-26T23:35:07.358 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.356+0000 7f53fbfff640 1 -- 172.21.15.139:0/1687573127 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f540401f030 con 0x7f54140b62c0 2024-07-26T23:35:07.358 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.356+0000 7f53fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:07.358 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.356+0000 7f53fbfff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:07.373 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.315+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-07-26T23:35:07.393 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.380+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 3 2024-07-26T23:35:07.393 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.392+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-07-26T23:35:07.453 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.412+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-07-26T23:35:07.454 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:07.452+0000 5dce500 -1 bdev(0x154c82e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-07-26T23:35:07.680 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.678+0000 7f5419cbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:07.680 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.678+0000 7f5419cbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:07.680 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.678+0000 7f5419cbc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:07.680 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.678+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5414076800 con 0x7f54140b62c0 2024-07-26T23:35:07.765 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.763+0000 7f53fbfff640 1 -- 172.21.15.139:0/1687573127 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+10434 (secure 0 0 0) 0x7f54040619c0 con 0x7f54140b62c0 2024-07-26T23:35:07.765 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.763+0000 7f53fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:07.765 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.763+0000 7f53fbfff640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:07.765 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:35:07.765 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":17,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:35:04.048498+0000","last_up_change":"2024-07-26T23:35:01.337925+0000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T23:35:00.571999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6803","nonce":228413488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6805","nonce":228413488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6809","nonce":228413488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6807","nonce":228413488}]},"public_addr":"172.21.15.177:6803/228413488","cluster_addr":"172.21.15.177:6805/228413488","heartbeat_back_addr":"172.21.15.177:6809/228413488","heartbeat_front_addr":"172.21.15.177:6807/228413488","state":["exists","up"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6811","nonce":1761005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6813","nonce":1761005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6817","nonce":1761005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6815","nonce":1761005298}]},"public_addr":"172.21.15.177:6811/1761005298","cluster_addr":"172.21.15.177:6813/1761005298","heartbeat_back_addr":"172.21.15.177:6817/1761005298","heartbeat_front_addr":"172.21.15.177:6815/1761005298","state":["exists","up"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6819","nonce":3299104561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6821","nonce":3299104561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6825","nonce":3299104561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6823","nonce":3299104561}]},"public_addr":"172.21.15.177:6819/3299104561","cluster_addr":"172.21.15.177:6821/3299104561","heartbeat_back_addr":"172.21.15.177:6825/3299104561","heartbeat_front_addr":"172.21.15.177:6823/3299104561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.331590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.423536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.831360+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:35:07.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.765+0000 7f5419cbc640 10 librados: watch_flush enter 2024-07-26T23:35:07.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.765+0000 7f5419cbc640 10 librados: watch_flush exit 2024-07-26T23:35:07.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.765+0000 7f5419cbc640 20 client.4258.objecter shutdown clearing up homeless session... 2024-07-26T23:35:07.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.765+0000 7f5419cbc640 10 client.4258.objecter successfully canceled tick 2024-07-26T23:35:07.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f54000789a0 msgr2=0x7f540007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:07.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f54000789a0 0x7f540007ae60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f54080094a0 tx=0x7f54080093f0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.768 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 10 monclient: shutdown 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54140b62c0 msgr2=0x7f541413f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54140b62c0 0x7f541413f490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f540400b3f0 tx=0x7f540400b8b0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 shutdown_connections 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f54000789a0 0x7f540007ae60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5414143970 0x7f541413b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5414144490 0x7f541413bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 --2- 172.21.15.139:0/1687573127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54140b62c0 0x7f541413f490 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 >> 172.21.15.139:0/1687573127 conn(0x7f541406e710 msgr2=0x7f54140b9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 shutdown_connections 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 -- 172.21.15.139:0/1687573127 wait complete. 2024-07-26T23:35:07.769 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:07.766+0000 7f5419cbc640 1 librados: shutdown 2024-07-26T23:35:07.778 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-07-26T23:35:08.474 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:08.469+0000 bc99640 -1 osd.4 0 waiting for initial osdmap 2024-07-26T23:35:08.911 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:08.886+0000 5dce500 -1 bdev(0x154c2b50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-07-26T23:35:08.956 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:35:08.942+0000 14269640 -1 osd.4 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:35:09.609 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:08.909+0000 5dce500 -1 bdev(0x154c2b50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-07-26T23:35:09.617 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:09.136+0000 5dce500 -1 bdev(0x154c2b50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-07-26T23:35:09.964 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:09.185+0000 5dce500 -1 bdev(0x154c2b50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-07-26T23:35:09.965 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:09.599+0000 5dce500 -1 bdev(0x154c2b50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-07-26T23:35:09.968 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:09.616+0000 5dce500 -1 bdev(0x154c2b50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-07-26T23:35:10.017 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.006+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.435 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.362+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.454 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.434+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.506 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.382+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.595 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.452+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.596 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.474+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.596 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.565+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.600 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.594+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.654 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.441+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.655 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.476+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.724 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.599+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.724 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:10.615+0000 5dce500 -1 bdev(0x17eaaa70 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-07-26T23:35:10.725 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.656+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.810 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.724+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.847 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.809+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.927 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.847+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.933 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:10.926+0000 5dce500 -1 bdev(0x154c9510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-07-26T23:35:10.955 INFO:tasks.ceph.osd.3.smithi139.stderr:2024-07-26T23:35:10.948+0000 b485640 -1 osd.3 0 waiting for initial osdmap 2024-07-26T23:35:11.048 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:11.041+0000 bc99640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T23:35:11.568 INFO:tasks.ceph.osd.3.smithi139.stderr:2024-07-26T23:35:11.545+0000 14269640 -1 osd.3 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:35:11.745 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:35:11.722+0000 14269640 -1 osd.1 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:35:11.896 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:19.567 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:11.987 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:19.649 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.042 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:19.705 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.044 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:19.715 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.092 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:19.764 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.159 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:12.152+0000 b485640 -1 osd.2 0 waiting for initial osdmap 2024-07-26T23:35:12.174 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:19.845 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.183 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:12.177+0000 bc99640 -1 osd.0 0 waiting for initial osdmap 2024-07-26T23:35:12.272 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:19.935 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.363 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:20.034 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.452 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:20.124 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:12.826 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:35:12.808+0000 14269640 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:35:12.852 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:35:12.836+0000 14269640 -1 osd.0 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T23:35:14.591 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.681+0000 7fee8b777640 10 monclient: get_monmap_and_config 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.681+0000 7fee8b777640 10 monclient: build_initial_monmap 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.681+0000 7fee8b777640 10 monclient: monmap: 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:35:14.682922+0000 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:35:14.682922+0000 2024-07-26T23:35:14.683 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:14.684 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:14.684 INFO:teuthology.misc.health.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:14.684 INFO:teuthology.misc.health.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:14.684 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:14.684 INFO:teuthology.misc.health.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:14.684 INFO:teuthology.misc.health.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:14.684 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.682+0000 7fee8b777640 1 Processor -- start 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.682+0000 7fee8b777640 1 -- start start 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.682+0000 7fee8b777640 10 monclient: init 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fee840b62c0 0x7fee84143970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient(hunting): picked mon.noname-a con 0x7fee840b62c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee840b66a0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fee84144490 0x7fee84139df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:14.687 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient(hunting): picked mon.noname-c con 0x7fee84144490 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee840acc10 con 0x7fee840b66a0 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fee84144920 con 0x7fee840b62c0 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fee840b4960 con 0x7fee84144490 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee8b777640 10 monclient(hunting): authenticate will time out at 1347.719330s 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.683+0000 7fee89513640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fee840b62c0 0x7fee84143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89d14640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fee84144490 0x7fee84139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89d14640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fee84144490 0x7fee84139df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:53958/0 (socket says 172.21.15.139:53958) 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89d14640 1 -- 172.21.15.139:0/3034270650 learned_addr learned my addr 172.21.15.139:0/3034270650 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89d14640 10 monclient(hunting): get_auth_request con 0x7fee84144490 auth_method 0 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89d14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:14.688 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89d14640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89d14640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89513640 10 monclient(hunting): get_auth_request con 0x7fee840b62c0 auth_method 0 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89513640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89513640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.684+0000 7fee89513640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.685+0000 7fee89d14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.685+0000 7fee89d14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.685+0000 7fee89d14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.685+0000 7fee89513640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.685+0000 7fee89513640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:14.689 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.685+0000 7fee89513640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:14.690 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-07-26T23:35:14.690 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 1 -- 172.21.15.139:0/3034270650 >> v2:172.21.15.139:3300/0 conn(0x7fee840b62c0 msgr2=0x7fee84143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:14.690 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 1 --2- 172.21.15.139:0/3034270650 >> v2:172.21.15.139:3300/0 conn(0x7fee840b62c0 0x7fee84143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:14.690 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 1 -- 172.21.15.139:0/3034270650 >> v1:172.21.15.139:6789/0 conn(0x7fee840b66a0 legacy=0x7fee840b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:14.691 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 10 monclient: _finish_hunting 0 2024-07-26T23:35:14.691 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 1 monclient: found mon.noname-c 2024-07-26T23:35:14.691 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:14.691 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:14.691 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.686+0000 7fee89d14640 1 -- 172.21.15.139:0/3034270650 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8413a330 con 0x7fee84144490 2024-07-26T23:35:14.691 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.689+0000 7fee89d14640 1 --2- 172.21.15.139:0/3034270650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee84144490 0x7fee84139df0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fee840acfc0 tx=0x7fee7c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce82b0b8923a2b7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:14.694 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 1 -- 172.21.15.139:0/3034270650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c02c9f0 con 0x7fee84144490 2024-07-26T23:35:14.694 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:14.694 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:14.694 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: dump: 2024-07-26T23:35:14.694 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:14.694 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: _finish_auth 0 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 1 -- 172.21.15.139:0/3034270650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee7c02cb90 con 0x7fee84144490 2024-07-26T23:35:14.695 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 1 -- 172.21.15.139:0/3034270650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0343f0 con 0x7fee84144490 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee8b777640 5 monclient: authenticate success, global_id 4261 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee8b777640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee8b777640 4 set_mon_vals no callback set 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.692+0000 7fee7bfff640 10 monclient: dump: 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:14.696 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:14.697 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:14.697 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:14.697 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:14.697 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:14.700 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 10 monclient: shutdown 2024-07-26T23:35:14.700 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:14.700 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 -- 172.21.15.139:0/3034270650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee84144490 msgr2=0x7fee84139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:14.700 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 --2- 172.21.15.139:0/3034270650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee84144490 0x7fee84139df0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fee840acfc0 tx=0x7fee7c02be50 comp rx=0 tx=0).stop 2024-07-26T23:35:14.700 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 -- 172.21.15.139:0/3034270650 shutdown_connections 2024-07-26T23:35:14.700 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 --2- 172.21.15.139:0/3034270650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee84144490 0x7fee84139df0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 --2- 172.21.15.139:0/3034270650 >> v2:172.21.15.139:3300/0 conn(0x7fee840b62c0 0x7fee84143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 -- 172.21.15.139:0/3034270650 >> 172.21.15.139:0/3034270650 conn(0x7fee8406e710 msgr2=0x7fee8402d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 -- 172.21.15.139:0/3034270650 shutdown_connections 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.693+0000 7fee8b777640 1 -- 172.21.15.139:0/3034270650 wait complete. 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 10 monclient: build_initial_monmap 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 10 monclient: monmap: 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:epoch 0 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:created 0.000000 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:14.701 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 1 librados: starting msgr at 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 1 librados: starting objecter 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 1 Processor -- start 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 1 -- start start 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 1 librados: setting wanted keys 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 1 librados: calling monclient init 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.694+0000 7fee8b777640 10 monclient: init 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:14.702 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee840b62c0 0x7fee8413d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): picked mon.noname-b con 0x7fee840b62c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee840b66a0 0x7fee8413aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): picked mon.noname-a con 0x7fee840b66a0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fee84144490 0x7fee8413af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): picked mon.noname-c con 0x7fee84144490 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fee840b6ac0 con 0x7fee840b66a0 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee8413a860 con 0x7fee840b62c0 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fee8414e280 con 0x7fee84144490 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:14.703 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee8b777640 10 monclient(hunting): authenticate will time out at 1347.731372s 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee89513640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee840b62c0 0x7fee8413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.695+0000 7fee89d14640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fee84144490 0x7fee8413af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee88d12640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee840b66a0 0x7fee8413aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee840b62c0 0x7fee8413d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:53960/0 (socket says 172.21.15.139:53960) 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 1 -- 172.21.15.139:0/3931925839 learned_addr learned my addr 172.21.15.139:0/3931925839 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 10 monclient(hunting): get_auth_request con 0x7fee840b62c0 auth_method 0 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89d14640 10 monclient(hunting): get_auth_request con 0x7fee84144490 auth_method 0 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89d14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89d14640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89d14640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:14.704 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee88d12640 10 monclient(hunting): get_auth_request con 0x7fee840b66a0 auth_method 0 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee88d12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee88d12640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee88d12640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.696+0000 7fee89513640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.697+0000 7fee89d14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.697+0000 7fee89d14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.697+0000 7fee89d14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.697+0000 7fee88d12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.697+0000 7fee88d12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:14.705 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.697+0000 7fee88d12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:14.707 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-07-26T23:35:14.707 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 1 -- 172.21.15.139:0/3931925839 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fee84144490 msgr2=0x7fee8413af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fee84144490 0x7fee8413af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 1 -- 172.21.15.139:0/3931925839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee840b66a0 msgr2=0x7fee8413aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee840b66a0 0x7fee8413aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 10 monclient: _finish_hunting 0 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 1 monclient: found mon.noname-b 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.699+0000 7fee89513640 1 -- 172.21.15.139:0/3931925839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8413c8b0 con 0x7fee840b62c0 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.702+0000 7fee89513640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee840b62c0 0x7fee8413d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e9b0 tx=0x7fee8000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.702+0000 7fee79ffb640 10 client.?.objecter ms_handle_connect 0x7fee840b62c0 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.702+0000 7fee79ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 1 -- 172.21.15.139:0/3931925839 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80017070 con 0x7fee840b62c0 2024-07-26T23:35:14.708 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: dump: 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:14.709 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 1 -- 172.21.15.139:0/3931925839 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee8000c9a0 con 0x7fee840b62c0 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 5 monclient: authenticate success, global_id 4264 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 1 -- 172.21.15.139:0/3931925839 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8000cd90 con 0x7fee840b62c0 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 10 monclient: _renew_subs 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee8413cb90 con 0x7fee840b62c0 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee79ffb640 10 monclient: dump: 2024-07-26T23:35:14.710 INFO:teuthology.misc.health.smithi139.stderr:epoch 1 2024-07-26T23:35:14.711 INFO:teuthology.misc.health.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:14.711 INFO:teuthology.misc.health.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.711 INFO:teuthology.misc.health.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:14.711 INFO:teuthology.misc.health.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:14.711 INFO:teuthology.misc.health.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:14.711 INFO:teuthology.misc.health.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr: 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee7b7fe640 4 set_mon_vals no callback set 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 10 monclient: _renew_subs 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.705+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee8413bba0 con 0x7fee840b62c0 2024-07-26T23:35:14.712 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.706+0000 7fee8b777640 1 librados: init done 2024-07-26T23:35:14.713 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.706+0000 7fee8b777640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:14.713 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.706+0000 7fee8b777640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:14.713 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.706+0000 7fee8b777640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:14.713 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.706+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee84143970 con 0x7fee840b62c0 2024-07-26T23:35:14.975 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.973+0000 7fee79ffb640 1 -- 172.21.15.139:0/3931925839 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fee8001a070 con 0x7fee840b62c0 2024-07-26T23:35:14.975 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee79ffb640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fee680789d0 0x7fee6807ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee79ffb640 1 -- 172.21.15.139:0/3931925839 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4857+0+0 (secure 0 0 0) 0x7fee80068550 con 0x7fee840b62c0 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee79ffb640 10 client.4264.objecter ms_dispatch 0x7fee84000b90 osd_map(21..21 src has 1..21) 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee79ffb640 3 client.4264.objecter handle_osd_map got epochs [21,21] > 0 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee79ffb640 3 client.4264.objecter handle_osd_map decoding full epoch 21 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee88d12640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fee680789d0 0x7fee6807ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee79ffb640 20 client.4264.objecter dump_active .. 0 homeless 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.974+0000 7fee88d12640 10 monclient: get_auth_request con 0x7fee680789d0 auth_method 0 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.975+0000 7fee79ffb640 10 client.4264.objecter ms_handle_connect 0x7fee680789d0 2024-07-26T23:35:14.976 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.975+0000 7fee88d12640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fee680789d0 0x7fee6807ae90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee8413dc10 tx=0x7fee74009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:14.988 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.986+0000 7fee79ffb640 1 -- 172.21.15.139:0/3931925839 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fee80061b10 con 0x7fee840b62c0 2024-07-26T23:35:14.988 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.986+0000 7fee79ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:14.988 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:14.986+0000 7fee79ffb640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:15.192 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.191+0000 7fee8b777640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:15.214 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.191+0000 7fee8b777640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:15.214 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.191+0000 7fee8b777640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:15.214 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.191+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee8413cd60 con 0x7fee840b62c0 2024-07-26T23:35:15.312 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.311+0000 7fee79ffb640 1 -- 172.21.15.139:0/3931925839 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+12909 (secure 0 0 0) 0x7fee80066060 con 0x7fee840b62c0 2024-07-26T23:35:15.312 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.311+0000 7fee79ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:15.313 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.311+0000 7fee79ffb640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:15.313 INFO:teuthology.misc.health.smithi139.stdout: 2024-07-26T23:35:15.313 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":22,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:35:15.058884+0000","last_up_change":"2024-07-26T23:35:14.011640+0000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T23:35:00.571999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6809","nonce":2123669081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6811","nonce":2123669081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6815","nonce":2123669081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6813","nonce":2123669081}]},"public_addr":"172.21.15.139:6809/2123669081","cluster_addr":"172.21.15.139:6811/2123669081","heartbeat_back_addr":"172.21.15.139:6815/2123669081","heartbeat_front_addr":"172.21.15.139:6813/2123669081","state":["exists","up"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6801","nonce":3107328002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6803","nonce":3107328002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6807","nonce":3107328002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6805","nonce":3107328002}]},"public_addr":"172.21.15.139:6801/3107328002","cluster_addr":"172.21.15.139:6803/3107328002","heartbeat_back_addr":"172.21.15.139:6807/3107328002","heartbeat_front_addr":"172.21.15.139:6805/3107328002","state":["exists","up"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6825","nonce":1043169956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6827","nonce":1043169956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6831","nonce":1043169956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6829","nonce":1043169956}]},"public_addr":"172.21.15.139:6825/1043169956","cluster_addr":"172.21.15.139:6827/1043169956","heartbeat_back_addr":"172.21.15.139:6831/1043169956","heartbeat_front_addr":"172.21.15.139:6829/1043169956","state":["exists","up"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6817","nonce":4033735752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6819","nonce":4033735752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6823","nonce":4033735752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6821","nonce":4033735752}]},"public_addr":"172.21.15.139:6817/4033735752","cluster_addr":"172.21.15.139:6819/4033735752","heartbeat_back_addr":"172.21.15.139:6823/4033735752","heartbeat_front_addr":"172.21.15.139:6821/4033735752","state":["exists","up"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6827","nonce":4112782318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6829","nonce":4112782318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6833","nonce":4112782318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6831","nonce":4112782318}]},"public_addr":"172.21.15.177:6827/4112782318","cluster_addr":"172.21.15.177:6829/4112782318","heartbeat_back_addr":"172.21.15.177:6833/4112782318","heartbeat_front_addr":"172.21.15.177:6831/4112782318","state":["exists","up"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6803","nonce":228413488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6805","nonce":228413488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6809","nonce":228413488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6807","nonce":228413488}]},"public_addr":"172.21.15.177:6803/228413488","cluster_addr":"172.21.15.177:6805/228413488","heartbeat_back_addr":"172.21.15.177:6809/228413488","heartbeat_front_addr":"172.21.15.177:6807/228413488","state":["exists","up"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6811","nonce":1761005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6813","nonce":1761005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6817","nonce":1761005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6815","nonce":1761005298}]},"public_addr":"172.21.15.177:6811/1761005298","cluster_addr":"172.21.15.177:6813/1761005298","heartbeat_back_addr":"172.21.15.177:6817/1761005298","heartbeat_front_addr":"172.21.15.177:6815/1761005298","state":["exists","up"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6819","nonce":3299104561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6821","nonce":3299104561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6825","nonce":3299104561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6823","nonce":3299104561}]},"public_addr":"172.21.15.177:6819/3299104561","cluster_addr":"172.21.15.177:6821/3299104561","heartbeat_back_addr":"172.21.15.177:6825/3299104561","heartbeat_front_addr":"172.21.15.177:6823/3299104561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.190400+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:35:00.557599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.842601+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:58.007938+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.331590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.423536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.831360+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 10 librados: watch_flush enter 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 10 librados: watch_flush exit 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 20 client.4264.objecter shutdown clearing up homeless session... 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 10 client.4264.objecter successfully canceled tick 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fee680789d0 msgr2=0x7fee6807ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fee680789d0 0x7fee6807ae90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee8413dc10 tx=0x7fee74009040 comp rx=0 tx=0).stop 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 10 monclient: shutdown 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:15.315 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee840b62c0 msgr2=0x7fee8413d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee840b62c0 0x7fee8413d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e9b0 tx=0x7fee8000ee70 comp rx=0 tx=0).stop 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 shutdown_connections 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fee680789d0 0x7fee6807ae90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fee84144490 0x7fee8413af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee840b66a0 0x7fee8413aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 --2- 172.21.15.139:0/3931925839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee840b62c0 0x7fee8413d600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 >> 172.21.15.139:0/3931925839 conn(0x7fee8406e710 msgr2=0x7fee84138820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 shutdown_connections 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 -- 172.21.15.139:0/3931925839 wait complete. 2024-07-26T23:35:15.316 INFO:teuthology.misc.health.smithi139.stderr:2024-07-26T23:35:15.313+0000 7fee8b777640 1 librados: shutdown 2024-07-26T23:35:15.324 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-07-26T23:35:15.324 INFO:tasks.ceph:Creating RBD pool 2024-07-26T23:35:15.324 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster ceph osd pool create rbd 8 2024-07-26T23:35:15.424 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.423+0000 7fad67309640 10 monclient: get_monmap_and_config 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.423+0000 7fad67309640 10 monclient: build_initial_monmap 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.423+0000 7fad67309640 10 monclient: monmap: 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:15.424676+0000 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:15.424676+0000 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:15.425 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:15.426 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:15.426 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:15.426 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:15.426 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 1 Processor -- start 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 1 -- start start 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 10 monclient: init 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fad60068740 0x7fad60140c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 10 monclient(hunting): picked mon.noname-a con 0x7fad60068740 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:15.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad601411d0 0x7fad60146f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.424+0000 7fad67309640 10 monclient(hunting): picked mon.noname-b con 0x7fad601411d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad60147480 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad600b0040 con 0x7fad60147480 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fad60068b20 con 0x7fad60068740 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fad6006da70 con 0x7fad601411d0 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad67309640 10 monclient(hunting): authenticate will time out at 1348.461040s 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad650a5640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fad60068740 0x7fad60140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad648a4640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad601411d0 0x7fad60146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:15.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad648a4640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad601411d0 0x7fad60146f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:55788/0 (socket says 172.21.15.139:55788) 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad648a4640 1 -- 172.21.15.139:0/539341185 learned_addr learned my addr 172.21.15.139:0/539341185 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad648a4640 10 monclient(hunting): get_auth_request con 0x7fad601411d0 auth_method 0 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad648a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad648a4640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.425+0000 7fad648a4640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad650a5640 10 monclient(hunting): get_auth_request con 0x7fad60068740 auth_method 0 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad650a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:15.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad650a5640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad650a5640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad648a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad648a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad648a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad650a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad650a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.426+0000 7fad650a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 1 -- 172.21.15.139:0/539341185 >> v2:172.21.15.139:3300/0 conn(0x7fad60068740 msgr2=0x7fad60140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 1 --2- 172.21.15.139:0/539341185 >> v2:172.21.15.139:3300/0 conn(0x7fad60068740 0x7fad60140c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 1 -- 172.21.15.139:0/539341185 >> v1:172.21.15.139:6789/0 conn(0x7fad60147480 legacy=0x7fad600b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 10 monclient: _finish_hunting 0 2024-07-26T23:35:15.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 1 monclient: found mon.noname-b 2024-07-26T23:35:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.428+0000 7fad648a4640 1 -- 172.21.15.139:0/539341185 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad601415f0 con 0x7fad601411d0 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.430+0000 7fad648a4640 1 --2- 172.21.15.139:0/539341185 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad601411d0 0x7fad60146f40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fad50009870 tx=0x7fad5002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9e24077e266c623 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 1 -- 172.21.15.139:0/539341185 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50035020 con 0x7fad601411d0 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: dump: 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:15.436 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.437 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.437 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:15.437 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:15.437 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:15.437 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:15.437 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:15.437 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 1 -- 172.21.15.139:0/539341185 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad50031640 con 0x7fad601411d0 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad67309640 5 monclient: authenticate success, global_id 4229 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad67309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad67309640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:15.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad67309640 4 set_mon_vals no callback set 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 1 -- 172.21.15.139:0/539341185 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5002d040 con 0x7fad601411d0 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.433+0000 7fad57fff640 10 monclient: dump: 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:15.439 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 10 monclient: shutdown 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 -- 172.21.15.139:0/539341185 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad601411d0 msgr2=0x7fad60146f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 --2- 172.21.15.139:0/539341185 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad601411d0 0x7fad60146f40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fad50009870 tx=0x7fad5002b8f0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 -- 172.21.15.139:0/539341185 shutdown_connections 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 --2- 172.21.15.139:0/539341185 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad601411d0 0x7fad60146f40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 --2- 172.21.15.139:0/539341185 >> v2:172.21.15.139:3300/0 conn(0x7fad60068740 0x7fad60140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 -- 172.21.15.139:0/539341185 >> 172.21.15.139:0/539341185 conn(0x7fad6006c720 msgr2=0x7fad6006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 -- 172.21.15.139:0/539341185 shutdown_connections 2024-07-26T23:35:15.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.434+0000 7fad67309640 1 -- 172.21.15.139:0/539341185 wait complete. 2024-07-26T23:35:15.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.435+0000 7fad67309640 10 monclient: build_initial_monmap 2024-07-26T23:35:15.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.435+0000 7fad67309640 10 monclient: monmap: 2024-07-26T23:35:15.445 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.435+0000 7fad67309640 1 librados: starting msgr at 2024-07-26T23:35:15.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.435+0000 7fad67309640 1 librados: starting objecter 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.435+0000 7fad67309640 1 Processor -- start 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 -- start start 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 librados: setting wanted keys 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 librados: calling monclient init 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient: init 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad60068740 0x7fad6011daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): picked mon.noname-b con 0x7fad60068740 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:15.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fad601411d0 0x7fad60141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): picked mon.noname-a con 0x7fad601411d0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad60147480 0x7fad60141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): picked mon.noname-c con 0x7fad60147480 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fad60147930 con 0x7fad601411d0 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad6006d2a0 con 0x7fad60068740 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fad600b02f0 con 0x7fad60147480 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.436+0000 7fad67309640 10 monclient(hunting): authenticate will time out at 1348.472654s 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad60147480 0x7fad60141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:15.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad648a4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fad601411d0 0x7fad60141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad60147480 0x7fad60141f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:55800/0 (socket says 172.21.15.139:55800) 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 1 -- 172.21.15.139:0/234871804 learned_addr learned my addr 172.21.15.139:0/234871804 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 10 monclient(hunting): get_auth_request con 0x7fad60147480 auth_method 0 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad650a5640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad60068740 0x7fad6011daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad648a4640 10 monclient(hunting): get_auth_request con 0x7fad601411d0 auth_method 0 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad648a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad648a4640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad648a4640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:15.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad650a5640 10 monclient(hunting): get_auth_request con 0x7fad60068740 auth_method 0 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad650a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad650a5640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad650a5640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.437+0000 7fad658a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.438+0000 7fad648a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.438+0000 7fad648a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.438+0000 7fad648a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.438+0000 7fad650a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.438+0000 7fad650a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:15.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.438+0000 7fad650a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 1 -- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad60068740 msgr2=0x7fad6011daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad60068740 0x7fad6011daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 1 -- 172.21.15.139:0/234871804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fad601411d0 msgr2=0x7fad60141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fad601411d0 0x7fad60141a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 10 monclient: _finish_hunting 0 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 1 monclient: found mon.noname-c 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.440+0000 7fad658a6640 1 -- 172.21.15.139:0/234871804 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6011ec00 con 0x7fad60147480 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.442+0000 7fad658a6640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad60147480 0x7fad60141f40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fad5800b9e0 tx=0x7fad5800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.442+0000 7fad55ffb640 10 client.?.objecter ms_handle_connect 0x7fad60147480 2024-07-26T23:35:15.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.442+0000 7fad55ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:15.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 1 -- 172.21.15.139:0/234871804 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58003b70 con 0x7fad60147480 2024-07-26T23:35:15.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:15.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:15.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: dump: 2024-07-26T23:35:15.453 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:15.454 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 1 -- 172.21.15.139:0/234871804 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad58003d10 con 0x7fad60147480 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 5 monclient: authenticate success, global_id 4232 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 1 -- 172.21.15.139:0/234871804 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5800cae0 con 0x7fad60147480 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 10 monclient: _renew_subs 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:15.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6011eee0 con 0x7fad60147480 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad55ffb640 10 monclient: dump: 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:15.456 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad577fe640 4 set_mon_vals no callback set 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 10 monclient: _renew_subs 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fad601443f0 con 0x7fad60147480 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.446+0000 7fad67309640 1 librados: init done 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.447+0000 7fad67309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.447+0000 7fad67309640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.447+0000 7fad67309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:15.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.447+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad6006cb00 con 0x7fad60147480 2024-07-26T23:35:15.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.468+0000 7fad55ffb640 1 -- 172.21.15.139:0/234871804 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fad5800cc80 con 0x7fad60147480 2024-07-26T23:35:15.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.469+0000 7fad55ffb640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fad440789a0 0x7fad4407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:15.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.469+0000 7fad650a5640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fad440789a0 0x7fad4407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:15.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.469+0000 7fad650a5640 10 monclient: get_auth_request con 0x7fad440789a0 auth_method 0 2024-07-26T23:35:15.471 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.470+0000 7fad650a5640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fad440789a0 0x7fad4407ae60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad60148220 tx=0x7fad5c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:15.471 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.470+0000 7fad55ffb640 10 client.4232.objecter ms_handle_connect 0x7fad440789a0 2024-07-26T23:35:15.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 1 -- 172.21.15.139:0/234871804 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 4857+0+0 (secure 0 0 0) 0x7fad58068380 con 0x7fad60147480 2024-07-26T23:35:15.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 10 client.4232.objecter ms_dispatch 0x7fad60000b90 osd_map(22..22 src has 1..22) 2024-07-26T23:35:15.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 3 client.4232.objecter handle_osd_map got epochs [22,22] > 0 2024-07-26T23:35:15.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 3 client.4232.objecter handle_osd_map decoding full epoch 22 2024-07-26T23:35:15.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 20 client.4232.objecter dump_active .. 0 homeless 2024-07-26T23:35:15.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 1 -- 172.21.15.139:0/234871804 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fad6006cb00 con 0x7fad60147480 2024-07-26T23:35:15.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:15.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.711+0000 7fad55ffb640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:15.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.918+0000 7fad67309640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-07-26T23:35:15.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.918+0000 7fad67309640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-07-26T23:35:15.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.918+0000 7fad67309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:15.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:15.918+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fad6006fcf0 con 0x7fad60147480 2024-07-26T23:35:16.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.355+0000 7fad55ffb640 1 -- 172.21.15.139:0/234871804 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) ==== 109+0+0 (secure 0 0 0) 0x7fad5806a1d0 con 0x7fad60147480 2024-07-26T23:35:16.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.355+0000 7fad55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-07-26T23:35:16.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.355+0000 7fad55ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-07-26T23:35:16.357 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'rbd' created 2024-07-26T23:35:16.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 10 librados: watch_flush enter 2024-07-26T23:35:16.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 10 librados: watch_flush exit 2024-07-26T23:35:16.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 20 client.4232.objecter shutdown clearing up homeless session... 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 10 client.4232.objecter successfully canceled tick 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fad440789a0 msgr2=0x7fad4407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fad440789a0 0x7fad4407ae60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad60148220 tx=0x7fad5c009190 comp rx=0 tx=0).stop 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 10 monclient: shutdown 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad60147480 msgr2=0x7fad60141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad60147480 0x7fad60141f40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fad5800b9e0 tx=0x7fad5800bea0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 shutdown_connections 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fad440789a0 0x7fad4407ae60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fad60147480 0x7fad60141f40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fad601411d0 0x7fad60141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 --2- 172.21.15.139:0/234871804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad60068740 0x7fad6011daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.358+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 >> 172.21.15.139:0/234871804 conn(0x7fad6006c720 msgr2=0x7fad60072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:16.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.359+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 shutdown_connections 2024-07-26T23:35:16.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.359+0000 7fad67309640 1 -- 172.21.15.139:0/234871804 wait complete. 2024-07-26T23:35:16.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.359+0000 7fad67309640 1 librados: shutdown 2024-07-26T23:35:16.374 DEBUG:teuthology.orchestra.run.smithi139:> rbd --cluster ceph pool init rbd 2024-07-26T23:35:16.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.439+0000 7fb7978aaf00 10 monclient: get_monmap_and_config 2024-07-26T23:35:16.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.440+0000 7fb7978aaf00 10 monclient: build_initial_monmap 2024-07-26T23:35:16.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.440+0000 7fb7978aaf00 10 monclient: monmap: 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:16.441132+0000 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:16.441132+0000 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:16.442 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.441+0000 7fb7978aaf00 1 Processor -- start 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.441+0000 7fb7978aaf00 1 -- start start 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.441+0000 7fb7978aaf00 10 monclient: init 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient: _reopen_session rank -1 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient(hunting): picked mon.noname-c-legacy con 0x55b700918390 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 1 --2- >> v2:172.21.15.139:3300/0 conn(0x55b7008e7960 0x55b7008e6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient(hunting): picked mon.noname-a con 0x55b7008e7960 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b7008e6f50 0x55b7009a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:16.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient(hunting): picked mon.noname-b con 0x55b7008e6f50 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:16.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b70075ab50 con 0x55b700918390 2024-07-26T23:35:16.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:16.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x55b7007cf960 con 0x55b7008e7960 2024-07-26T23:35:16.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:16.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x55b7007e0460 con 0x55b7008e6f50 2024-07-26T23:35:16.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient(hunting): _renew_subs 2024-07-26T23:35:16.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.442+0000 7fb7978aaf00 10 monclient(hunting): authenticate will time out at 1349.478264s 2024-07-26T23:35:16.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b7008e6f50 0x55b7009a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:16.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b7008e6f50 0x55b7009a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:55816/0 (socket says 172.21.15.139:55816) 2024-07-26T23:35:16.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 1 -- 172.21.15.139:0/1325153060 learned_addr learned my addr 172.21.15.139:0/1325153060 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:16.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 10 monclient(hunting): get_auth_request con 0x55b7008e6f50 auth_method 0 2024-07-26T23:35:16.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:16.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:16.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:16.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:16.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:16.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.460+0000 7fb7975ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:16.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb795b9c640 1 --2- 172.21.15.139:0/1325153060 >> v2:172.21.15.139:3300/0 conn(0x55b7008e7960 0x55b7008e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:16.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-07-26T23:35:16.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 1 -- 172.21.15.139:0/1325153060 >> v2:172.21.15.139:3300/0 conn(0x55b7008e7960 msgr2=0x55b7008e6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:16.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 1 --2- 172.21.15.139:0/1325153060 >> v2:172.21.15.139:3300/0 conn(0x55b7008e7960 0x55b7008e6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 1 -- 172.21.15.139:0/1325153060 >> v1:172.21.15.177:6789/0 conn(0x55b700918390 legacy=0x55b7009a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:16.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:16.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 1 monclient: found mon.noname-b 2024-07-26T23:35:16.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:16.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:16.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.462+0000 7fb7975ff640 1 -- 172.21.15.139:0/1325153060 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7007748a0 con 0x55b7008e6f50 2024-07-26T23:35:16.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.467+0000 7fb7975ff640 1 --2- 172.21.15.139:0/1325153060 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b7008e6f50 0x55b7009a0e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb78800a560 tx=0x7fb78801cd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39725af91f9cb6d9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:16.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb79539b640 1 -- 172.21.15.139:0/1325153060 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb788052610 con 0x55b7008e6f50 2024-07-26T23:35:16.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb79539b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:16.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb79539b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:16.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb79539b640 10 monclient: dump: 2024-07-26T23:35:16.472 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:16.473 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:16.473 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.473 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.473 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:16.473 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:16.473 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:16.474 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:16.474 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:16.474 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:16.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb79539b640 10 monclient: _finish_auth 0 2024-07-26T23:35:16.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb79539b640 10 monclient: _check_auth_tickets 2024-07-26T23:35:16.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb79539b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:16.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb7978aaf00 5 monclient: authenticate success, global_id 4235 2024-07-26T23:35:16.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.470+0000 7fb7978aaf00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:16.480 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.471+0000 7fb79539b640 1 -- 172.21.15.139:0/1325153060 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb78804e070 con 0x55b7008e6f50 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.471+0000 7fb79539b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.471+0000 7fb7978aaf00 10 monclient: get_monmap_and_config success 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.471+0000 7fb7978aaf00 4 set_mon_vals no callback set 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb79539b640 1 -- 172.21.15.139:0/1325153060 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7880529f0 con 0x55b7008e6f50 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb79539b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb79539b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb79539b640 10 monclient: dump: 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.481 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 10 monclient: shutdown 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 1 -- 172.21.15.139:0/1325153060 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b7008e6f50 msgr2=0x55b7009a0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 1 --2- 172.21.15.139:0/1325153060 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b7008e6f50 0x55b7009a0e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb78800a560 tx=0x7fb78801cd40 comp rx=0 tx=0).stop 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 1 -- 172.21.15.139:0/1325153060 shutdown_connections 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 1 --2- 172.21.15.139:0/1325153060 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b7008e6f50 0x55b7009a0e40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 1 --2- 172.21.15.139:0/1325153060 >> v2:172.21.15.139:3300/0 conn(0x55b7008e7960 0x55b7008e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.472+0000 7fb7978aaf00 1 -- 172.21.15.139:0/1325153060 >> 172.21.15.139:0/1325153060 conn(0x55b7008e51b0 msgr2=0x55b7009179e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.473+0000 7fb7978aaf00 1 -- 172.21.15.139:0/1325153060 shutdown_connections 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.473+0000 7fb7978aaf00 1 -- 172.21.15.139:0/1325153060 wait complete. 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.473+0000 7fb7978aaf00 10 monclient: build_initial_monmap 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.473+0000 7fb7978aaf00 10 monclient: monmap: 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:16.483 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.474+0000 7fb7978aaf00 1 librados: starting msgr at 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.474+0000 7fb7978aaf00 1 librados: starting objecter 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.474+0000 7fb7978aaf00 1 Processor -- start 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.474+0000 7fb7978aaf00 1 -- start start 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.474+0000 7fb7978aaf00 1 librados: setting wanted keys 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.474+0000 7fb7978aaf00 1 librados: calling monclient init 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.474+0000 7fb7978aaf00 10 monclient: init 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient: _reopen_session rank -1 2024-07-26T23:35:16.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b7008e6f50 0x55b700910270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): picked mon.noname-a con 0x55b7008e6f50 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7008e7960 0x55b700910df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): picked mon.noname-b con 0x55b7008e7960 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b700918390 0x55b7009b9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): picked mon.noname-c con 0x55b700918390 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55b700775a70 con 0x55b7008e6f50 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b7007d39a0 con 0x55b7008e7960 2024-07-26T23:35:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x55b7009a1790 con 0x55b700918390 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): _renew_subs 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.475+0000 7fb7978aaf00 10 monclient(hunting): authenticate will time out at 1349.511609s 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb7975ff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b700918390 0x55b7009b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb79639d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b7008e6f50 0x55b700910270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb7975ff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b700918390 0x55b7009b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:55822/0 (socket says 172.21.15.139:55822) 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb7975ff640 1 -- 172.21.15.139:0/2846279691 learned_addr learned my addr 172.21.15.139:0/2846279691 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb7975ff640 10 monclient(hunting): get_auth_request con 0x55b700918390 auth_method 0 2024-07-26T23:35:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb7975ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb7975ff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb7975ff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb79639d640 10 monclient(hunting): get_auth_request con 0x55b7008e6f50 auth_method 0 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb79639d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb79639d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.476+0000 7fb79639d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.477+0000 7fb7975ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.477+0000 7fb7975ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.477+0000 7fb7975ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.477+0000 7fb79639d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.477+0000 7fb79639d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.477+0000 7fb79639d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 1 -- 172.21.15.139:0/2846279691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7008e7960 msgr2=0x55b700910df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7008e7960 0x55b700910df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 1 -- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b7008e6f50 msgr2=0x55b700910270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55b7008e6f50 0x55b700910270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 1 monclient: found mon.noname-c 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:35:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.479+0000 7fb7975ff640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b70074f0c0 con 0x55b700918390 2024-07-26T23:35:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.482+0000 7fb7975ff640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b700918390 0x55b7009b9060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb78801ba30 tx=0x7fb788003330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.482+0000 7fb7867fc640 10 client.?.objecter ms_handle_connect 0x55b700918390 2024-07-26T23:35:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.482+0000 7fb7867fc640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb788061020 con 0x55b700918390 2024-07-26T23:35:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: dump: 2024-07-26T23:35:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.493 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:16.493 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:16.493 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:16.493 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:16.493 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:16.493 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:16.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:16.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:16.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:16.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb788058040 con 0x55b700918390 2024-07-26T23:35:16.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:16.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb788005b40 con 0x55b700918390 2024-07-26T23:35:16.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:16.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:16.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7867fc640 10 monclient: dump: 2024-07-26T23:35:16.495 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:16.495 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:16.495 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.496 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:16.496 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:16.496 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:16.496 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:16.496 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:16.497 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:16.497 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:16.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 5 monclient: authenticate success, global_id 4238 2024-07-26T23:35:16.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 10 monclient: _renew_subs 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b7009a8ae0 con 0x55b700918390 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb794b9a640 4 set_mon_vals no callback set 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 10 monclient: _renew_subs 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55b7009a83a0 con 0x55b700918390 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 1 librados: init done 2024-07-26T23:35:16.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.487+0000 7fb7978aaf00 10 librados: wait_for_osdmap waiting 2024-07-26T23:35:16.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.525+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fb788064020 con 0x55b700918390 2024-07-26T23:35:16.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.526+0000 7fb7867fc640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb774076b50 0x7fb774079010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:16.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.526+0000 7fb79639d640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb774076b50 0x7fb774079010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:16.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.526+0000 7fb79639d640 10 monclient: get_auth_request con 0x7fb774076b50 auth_method 0 2024-07-26T23:35:16.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.527+0000 7fb79639d640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb774076b50 0x7fb774079010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00d620 tx=0x7fb78c0058a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:16.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.527+0000 7fb7867fc640 10 client.4238.objecter ms_handle_connect 0x7fb774076b50 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5199+0+0 (secure 0 0 0) 0x7fb78809e170 con 0x55b700918390 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7867fc640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_map(23..23 src has 1..23) 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map got epochs [23,23] > 0 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map decoding full epoch 23 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7867fc640 20 client.4238.objecter dump_active .. 0 homeless 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7978aaf00 10 librados: wait_for_osdmap done waiting 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7978aaf00 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7978aaf00 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-07-26T23:35:16.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7978aaf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:16.536 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:16.531+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55b70099fb00 con 0x55b700918390 2024-07-26T23:35:17.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.548+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v24) ==== 141+0+0 (secure 0 0 0) 0x7fb78806b030 con 0x55b700918390 2024-07-26T23:35:17.549 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.548+0000 7fb7867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-07-26T23:35:17.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.548+0000 7fb7867fc640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-07-26T23:35:17.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.548+0000 7fb7978aaf00 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:17.550 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.548+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb78804e070 con 0x55b700918390 2024-07-26T23:35:17.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.579+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb78809e530 con 0x55b700918390 2024-07-26T23:35:17.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.579+0000 7fb7867fc640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-07-26T23:35:17.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.579+0000 7fb794b9a640 10 client.4238.objecter _get_latest_version latest 24, waiting 2024-07-26T23:35:17.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.579+0000 7fb794b9a640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:17.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.579+0000 7fb794b9a640 10 monclient: _renew_subs 2024-07-26T23:35:17.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.579+0000 7fb794b9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:17.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.579+0000 7fb794b9a640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=24}) -- 0x7fb778003c90 con 0x55b700918390 2024-07-26T23:35:17.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 8 ==== osd_map(24..24 src has 1..24) ==== 658+0+0 (secure 0 0 0) 0x7fb7880aaa70 con 0x55b700918390 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7867fc640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_map(24..24 src has 1..24) 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map got epochs [24,24] > 23 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map decoding incremental epoch 24 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7867fc640 20 client.4238.objecter dump_active .. 0 homeless 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 10 librados: create oid=rbd_trash nspace= 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 10 client.4238.objecter _op_submit op 0x55b70099dce0 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 20 client.4238.objecter _calc_target epoch 24 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 20 client.4238.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 10 client.4238.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [3,2] primary 3 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x55b70090eab0 0x55b70090eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 20 client.4238.objecter _get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:17.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 10 client.4238.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.3 2024-07-26T23:35:17.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 20 client.4238.objecter get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:17.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 15 client.4238.objecter _session_op_assign 3 1 2024-07-26T23:35:17.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 15 client.4238.objecter _send_op 1 to 2.3 on osd.3 2024-07-26T23:35:17.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x55b70090f410 con 0x55b70090eab0 2024-07-26T23:35:17.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:17.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.582+0000 7fb7978aaf00 5 client.4238.objecter 1 in flight 2024-07-26T23:35:17.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.716+0000 7fb795b9c640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x55b70090eab0 0x55b70090eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:17.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.727+0000 7fb795b9c640 10 monclient: get_auth_request con 0x55b70090eab0 auth_method 0 2024-07-26T23:35:17.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.747+0000 7fb7867fc640 10 client.4238.objecter ms_handle_connect 0x55b70090eab0 2024-07-26T23:35:17.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:17.747+0000 7fb795b9c640 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x55b70090eab0 0x55b70090eed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:19.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== osd.3 v2:172.21.15.139:6816/4033735752 1 ==== osd_map(25..25 src has 1..25) ==== 288+0+0 (crc 0 0 0) 0x7fb780009610 con 0x55b70090eab0 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_map(25..25 src has 1..25) 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map got epochs [25,25] > 24 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map decoding incremental epoch 25 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 10 client.4238.objecter checking op 1 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 20 client.4238.objecter _calc_target epoch 25 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 20 client.4238.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 20 client.4238.objecter dump_active .. 0 homeless 2024-07-26T23:35:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.376+0000 7fb7867fc640 20 client.4238.objecter 1 2.5a749323 osd.3 rbd_trash [create] 2024-07-26T23:35:19.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.475+0000 7fb7857fa640 10 monclient: tick 2024-07-26T23:35:19.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.475+0000 7fb7857fa640 10 monclient: _check_auth_tickets 2024-07-26T23:35:19.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.475+0000 7fb7857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:19.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.475+0000 7fb7857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:19.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 1 -- 172.21.15.139:0/2846279691 <== osd.3 v2:172.21.15.139:6816/4033735752 2 ==== osd_op_reply(1 rbd_trash [create] v25'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7fb780009910 con 0x55b70090eab0 2024-07-26T23:35:19.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_op_reply(1 rbd_trash [create] v25'1 uv1 ondisk = 0) 2024-07-26T23:35:19.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 10 client.4238.objecter in handle_osd_op_reply 2024-07-26T23:35:19.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 7 client.4238.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-07-26T23:35:19.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 10 client.4238.objecter op 0 rval 0 len 0 2024-07-26T23:35:19.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 15 client.4238.objecter handle_osd_op_reply completed tid 1 2024-07-26T23:35:19.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 15 client.4238.objecter _finish_op 1 2024-07-26T23:35:19.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:19.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 15 client.4238.objecter _session_op_remove 3 1 2024-07-26T23:35:19.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb795b9c640 5 client.4238.objecter 0 in flight 2024-07-26T23:35:19.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.623+0000 7fb7978aaf00 10 librados: Objecter returned from create r=0 2024-07-26T23:35:19.627 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 10 client.4238.objecter _op_submit op 0x55b700a14310 2024-07-26T23:35:19.627 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 20 client.4238.objecter _calc_target epoch 25 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 20 client.4238.objecter _get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.3 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 20 client.4238.objecter get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 15 client.4238.objecter _session_op_assign 3 2 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 15 client.4238.objecter _send_op 2 to 2.2 on osd.3 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25) -- 0x55b700a14980 con 0x55b70090eab0 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:19.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.626+0000 7fb7978aaf00 5 client.4238.objecter 1 in flight 2024-07-26T23:35:19.654 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 1 -- 172.21.15.139:0/2846279691 <== osd.3 v2:172.21.15.139:6816/4033735752 3 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780009910 con 0x55b70090eab0 2024-07-26T23:35:19.654 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:35:19.655 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 10 client.4238.objecter in handle_osd_op_reply 2024-07-26T23:35:19.655 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 7 client.4238.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-07-26T23:35:19.655 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 10 client.4238.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:35:19.655 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 10 client.4238.objecter op 0 rval 0 len 0 2024-07-26T23:35:19.655 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 15 client.4238.objecter handle_osd_op_reply completed tid 2 2024-07-26T23:35:19.655 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 15 client.4238.objecter _finish_op 2 2024-07-26T23:35:19.655 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 15 client.4238.objecter _session_op_remove 3 2 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb795b9c640 5 client.4238.objecter 0 in flight 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 10 client.4238.objecter _op_submit op 0x55b700a134a0 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 20 client.4238.objecter _calc_target epoch 25 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 20 client.4238.objecter _get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.3 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 20 client.4238.objecter get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 15 client.4238.objecter _session_op_assign 3 3 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 15 client.4238.objecter _send_op 3 to 2.2 on osd.3 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.652+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25) -- 0x55b700a13950 con 0x55b70090eab0 2024-07-26T23:35:19.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.653+0000 7fb7978aaf00 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:19.657 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.653+0000 7fb7978aaf00 5 client.4238.objecter 1 in flight 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 1 -- 172.21.15.139:0/2846279691 <== osd.3 v2:172.21.15.139:6816/4033735752 4 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780009910 con 0x55b70090eab0 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 10 client.4238.objecter in handle_osd_op_reply 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 7 client.4238.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 10 client.4238.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 10 client.4238.objecter op 0 rval 0 len 0 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 15 client.4238.objecter handle_osd_op_reply completed tid 3 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 15 client.4238.objecter _finish_op 3 2024-07-26T23:35:19.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:19.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 15 client.4238.objecter _session_op_remove 3 3 2024-07-26T23:35:19.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb795b9c640 5 client.4238.objecter 0 in flight 2024-07-26T23:35:19.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb787fff640 10 client.4238.objecter allocate_selfmanaged_snap; pool: 2 2024-07-26T23:35:19.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb787fff640 10 client.4238.objecter pool_op_submit 4 2024-07-26T23:35:19.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb787fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:19.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:19.662+0000 7fb787fff640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7fb76c0030d0 con 0x55b700918390 2024-07-26T23:35:20.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.475+0000 7fb7857fa640 10 monclient: tick 2024-07-26T23:35:20.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.475+0000 7fb7857fa640 10 monclient: _check_auth_tickets 2024-07-26T23:35:20.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.475+0000 7fb7857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:20.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v27) ==== 55+0+0 (secure 0 0 0) 0x7fb788014d70 con 0x55b700918390 2024-07-26T23:35:20.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 pool_op_reply(tid 4 (0) Success v27) 2024-07-26T23:35:20.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 10 client.4238.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v27) 2024-07-26T23:35:20.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 10 client.4238.objecter have request 4 at 0x7fb76c003060 Op: create unmanaged snap 2024-07-26T23:35:20.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 20 client.4238.objecter waiting for client to reach epoch 27 before calling back 2024-07-26T23:35:20.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:20.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 10 monclient: _renew_subs 2024-07-26T23:35:20.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:20.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=26}) -- 0x7fb77407d2d0 con 0x55b700918390 2024-07-26T23:35:20.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.748+0000 7fb7867fc640 10 client.4238.objecter done 2024-07-26T23:35:20.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 10 ==== osd_map(26..27 src has 1..27) ==== 1216+0+0 (secure 0 0 0) 0x7fb7880d7300 con 0x55b700918390 2024-07-26T23:35:20.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb7867fc640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_map(26..27 src has 1..27) 2024-07-26T23:35:20.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map got epochs [26,27] > 25 2024-07-26T23:35:20.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map decoding incremental epoch 26 2024-07-26T23:35:20.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map decoding incremental epoch 27 2024-07-26T23:35:20.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb7867fc640 20 client.4238.objecter dump_active .. 0 homeless 2024-07-26T23:35:20.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 20 librados: queue_aio_write 0x55b700a0f8a0 completion 0x55b70078fa60 write_seq 1 2024-07-26T23:35:20.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 10 client.4238.objecter _op_submit op 0x7fb778006380 2024-07-26T23:35:20.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 20 client.4238.objecter _calc_target epoch 27 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 20 client.4238.objecter _get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.3 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 20 client.4238.objecter get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 15 client.4238.objecter _session_op_assign 3 5 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 15 client.4238.objecter _send_op 5 to 2.2 on osd.3 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x7fb7780069b0 con 0x55b70090eab0 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:20.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.755+0000 7fb794b9a640 5 client.4238.objecter 1 in flight 2024-07-26T23:35:20.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 1 -- 172.21.15.139:0/2846279691 <== osd.3 v2:172.21.15.139:6816/4033735752 5 ==== osd_op_reply(5 rbd_info [create,write 0~8] v27'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fb780009910 con 0x55b70090eab0 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_op_reply(5 rbd_info [create,write 0~8] v27'1 uv1 ondisk = 0) 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 10 client.4238.objecter in handle_osd_op_reply 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 7 client.4238.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 10 client.4238.objecter op 0 rval 0 len 0 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 10 client.4238.objecter op 1 rval 0 len 0 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 15 client.4238.objecter handle_osd_op_reply completed tid 5 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 15 client.4238.objecter _finish_op 5 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 15 client.4238.objecter _session_op_remove 3 5 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 5 client.4238.objecter 0 in flight 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb795b9c640 20 librados: complete_aio_write 0x55b70078fa60 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb787fff640 10 client.4238.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-07-26T23:35:20.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb787fff640 10 client.4238.objecter pool_op_submit 6 2024-07-26T23:35:20.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb787fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:20.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:20.989+0000 7fb787fff640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v27) -- 0x7fb76c004520 con 0x55b700918390 2024-07-26T23:35:21.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:21.475+0000 7fb7857fa640 10 monclient: tick 2024-07-26T23:35:21.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:21.475+0000 7fb7857fa640 10 monclient: _check_auth_tickets 2024-07-26T23:35:21.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:21.475+0000 7fb7857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:21.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:21.488+0000 7fb786ffd640 10 client.4238.objecter tick 2024-07-26T23:35:22.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v28) ==== 43+0+0 (secure 0 0 0) 0x7fb7880179b0 con 0x55b700918390 2024-07-26T23:35:22.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 pool_op_reply(tid 6 (0) Success v28) 2024-07-26T23:35:22.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 10 client.4238.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v28) 2024-07-26T23:35:22.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 10 client.4238.objecter have request 6 at 0x7fb76c003060 Op: delete unmanaged snap 2024-07-26T23:35:22.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 20 client.4238.objecter waiting for client to reach epoch 28 before calling back 2024-07-26T23:35:22.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:22.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 10 monclient: _renew_subs 2024-07-26T23:35:22.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:22.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=28}) -- 0x7fb77407d2d0 con 0x55b700918390 2024-07-26T23:35:22.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.048+0000 7fb7867fc640 10 client.4238.objecter done 2024-07-26T23:35:22.058 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb7867fc640 1 -- 172.21.15.139:0/2846279691 <== mon.2 v2:172.21.15.139:3301/0 12 ==== osd_map(28..28 src has 1..28) ==== 654+0+0 (secure 0 0 0) 0x7fb7880d7500 con 0x55b700918390 2024-07-26T23:35:22.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb7867fc640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_map(28..28 src has 1..28) 2024-07-26T23:35:22.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map got epochs [28,28] > 27 2024-07-26T23:35:22.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb7867fc640 3 client.4238.objecter handle_osd_map decoding incremental epoch 28 2024-07-26T23:35:22.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb7867fc640 20 client.4238.objecter dump_active .. 0 homeless 2024-07-26T23:35:22.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 20 librados: queue_aio_write 0x55b700a0f8a0 completion 0x7fb76c001d60 write_seq 2 2024-07-26T23:35:22.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 10 client.4238.objecter _op_submit op 0x7fb778006380 2024-07-26T23:35:22.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 20 client.4238.objecter _calc_target epoch 28 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 20 client.4238.objecter _get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.3 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 20 client.4238.objecter get_session s=0x55b70099e190 osd=3 3 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 15 client.4238.objecter _session_op_assign 3 7 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 15 client.4238.objecter _send_op 7 to 2.2 on osd.3 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 1 -- 172.21.15.139:0/2846279691 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) -- 0x7fb778006d50 con 0x55b70090eab0 2024-07-26T23:35:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:22.061 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.056+0000 7fb794b9a640 5 client.4238.objecter 1 in flight 2024-07-26T23:35:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 1 -- 172.21.15.139:0/2846279691 <== osd.3 v2:172.21.15.139:6816/4033735752 6 ==== osd_op_reply(7 rbd_info [write 0~19] v28'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780009910 con 0x55b70090eab0 2024-07-26T23:35:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 10 client.4238.objecter ms_dispatch 0x55b7007e81f0 osd_op_reply(7 rbd_info [write 0~19] v28'2 uv2 ondisk = 0) 2024-07-26T23:35:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 10 client.4238.objecter in handle_osd_op_reply 2024-07-26T23:35:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 7 client.4238.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-07-26T23:35:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 10 client.4238.objecter op 0 rval 0 len 0 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 15 client.4238.objecter handle_osd_op_reply completed tid 7 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 15 client.4238.objecter _finish_op 7 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 4 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 15 client.4238.objecter _session_op_remove 3 7 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 5 client.4238.objecter 0 in flight 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.261+0000 7fb795b9c640 20 librados: complete_aio_write 0x7fb76c001d60 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 10 librados: watch_flush enter 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 10 librados: watch_flush exit 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 10 client.4238.objecter close_session for osd.3 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x55b70090eab0 msgr2=0x55b70090eed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x55b70090eab0 0x55b70090eed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 20 client.4238.objecter put_session s=0x55b70099e190 osd=3 1 2024-07-26T23:35:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 20 client.4238.objecter shutdown clearing up homeless session... 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 10 client.4238.objecter successfully canceled tick 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb774076b50 msgr2=0x7fb774079010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb774076b50 0x7fb774079010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00d620 tx=0x7fb78c0058a0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 10 monclient: shutdown 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b700918390 msgr2=0x55b7009b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 --2- 172.21.15.139:0/2846279691 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55b700918390 0x55b7009b9060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb78801ba30 tx=0x7fb788003330 comp rx=0 tx=0).stop 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7975ff640 1 -- 172.21.15.139:0/2846279691 reap_dead start 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 shutdown_connections 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 >> 172.21.15.139:0/2846279691 conn(0x55b7008e51b0 msgr2=0x55b7009161e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 shutdown_connections 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 -- 172.21.15.139:0/2846279691 wait complete. 2024-07-26T23:35:22.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.262+0000 7fb7978aaf00 1 librados: shutdown 2024-07-26T23:35:22.268 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-07-26T23:35:22.269 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-07-26T23:35:22.269 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-07-26T23:35:22.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.368+0000 7f933082c640 10 monclient: get_monmap_and_config 2024-07-26T23:35:22.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.368+0000 7f933082c640 10 monclient: build_initial_monmap 2024-07-26T23:35:22.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.368+0000 7f933082c640 10 monclient: monmap: 2024-07-26T23:35:22.369 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:22.369 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:22.369203+0000 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:22.369203+0000 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:22.370 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:22.371 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:22.371 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:22.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.368+0000 7f933082c640 1 Processor -- start 2024-07-26T23:35:22.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.368+0000 7f933082c640 1 -- start start 2024-07-26T23:35:22.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.368+0000 7f933082c640 10 monclient: init 2024-07-26T23:35:22.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:22.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-07-26T23:35:22.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f932c068a70 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:22.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f932c0b0fc0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:22.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f932c1411b0 0x7f932c14f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:22.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient(hunting): picked mon.noname-a con 0x7f932c1411b0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:22.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932c0b5a30 con 0x7f932c0b0fc0 2024-07-26T23:35:22.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932c0b2140 con 0x7f932c068a70 2024-07-26T23:35:22.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:22.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f932c068e50 con 0x7f932c1411b0 2024-07-26T23:35:22.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f933082c640 10 monclient(hunting): authenticate will time out at 1355.405451s 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.369+0000 7f932ad9d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f932c1411b0 0x7f932c14f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.370+0000 7f932ad9d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f932c1411b0 0x7f932c14f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:60764/0 (socket says 172.21.15.139:60764) 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.370+0000 7f932ad9d640 1 -- 172.21.15.139:0/2333872880 learned_addr learned my addr 172.21.15.139:0/2333872880 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.370+0000 7f932ad9d640 10 monclient(hunting): get_auth_request con 0x7f932c1411b0 auth_method 0 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.370+0000 7f932ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.370+0000 7f932ad9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.370+0000 7f932ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.371+0000 7f932ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.371+0000 7f932ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:22.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.371+0000 7f932ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:22.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.372+0000 7f932ad9d640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-07-26T23:35:22.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.372+0000 7f932ad9d640 1 -- 172.21.15.139:0/2333872880 >> v1:172.21.15.177:6789/0 conn(0x7f932c068a70 legacy=0x7f932c147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.373+0000 7f932ad9d640 1 -- 172.21.15.139:0/2333872880 >> v1:172.21.15.139:6789/0 conn(0x7f932c0b0fc0 legacy=0x7f932c140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.373+0000 7f932ad9d640 10 monclient: _finish_hunting 0 2024-07-26T23:35:22.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.373+0000 7f932ad9d640 1 monclient: found mon.noname-a 2024-07-26T23:35:22.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.373+0000 7f932ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:22.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.373+0000 7f932ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:22.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.373+0000 7f932ad9d640 1 -- 172.21.15.139:0/2333872880 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c0b2420 con 0x7f932c1411b0 2024-07-26T23:35:22.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.375+0000 7f932ad9d640 1 --2- 172.21.15.139:0/2333872880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c1411b0 0x7f932c14f510 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9324009e50 tx=0x7f932402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805890ceebb5d662 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:22.381 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 1 -- 172.21.15.139:0/2333872880 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932402cb20 con 0x7f932c1411b0 2024-07-26T23:35:22.382 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:22.382 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:22.382 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: dump: 2024-07-26T23:35:22.382 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:22.382 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:22.382 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.383 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.383 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:22.383 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:22.383 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:22.383 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:22.383 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:22.384 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:22.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: _finish_auth 0 2024-07-26T23:35:22.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:35:22.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:22.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 1 -- 172.21.15.139:0/2333872880 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f932402ccc0 con 0x7f932c1411b0 2024-07-26T23:35:22.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f933082c640 5 monclient: authenticate success, global_id 4281 2024-07-26T23:35:22.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f933082c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:22.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:22.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f933082c640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:22.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 1 -- 172.21.15.139:0/2333872880 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9324034610 con 0x7f932c1411b0 2024-07-26T23:35:22.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:22.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f933082c640 4 set_mon_vals no callback set 2024-07-26T23:35:22.386 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:22.386 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.380+0000 7f9329d9b640 10 monclient: dump: 2024-07-26T23:35:22.386 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:22.386 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:22.387 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.387 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.387 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:22.387 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:22.387 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:22.388 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:22.388 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:22.388 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:22.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 10 monclient: shutdown 2024-07-26T23:35:22.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:22.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 1 -- 172.21.15.139:0/2333872880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c1411b0 msgr2=0x7f932c14f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 1 --2- 172.21.15.139:0/2333872880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c1411b0 0x7f932c14f510 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9324009e50 tx=0x7f932402bf80 comp rx=0 tx=0).stop 2024-07-26T23:35:22.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 1 -- 172.21.15.139:0/2333872880 shutdown_connections 2024-07-26T23:35:22.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 1 --2- 172.21.15.139:0/2333872880 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c1411b0 0x7f932c14f510 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 1 -- 172.21.15.139:0/2333872880 >> 172.21.15.139:0/2333872880 conn(0x7f932c0b6670 msgr2=0x7f932c0b6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:22.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 1 -- 172.21.15.139:0/2333872880 shutdown_connections 2024-07-26T23:35:22.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.381+0000 7f933082c640 1 -- 172.21.15.139:0/2333872880 wait complete. 2024-07-26T23:35:22.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.382+0000 7f933082c640 10 monclient: build_initial_monmap 2024-07-26T23:35:22.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.382+0000 7f933082c640 10 monclient: monmap: 2024-07-26T23:35:22.399 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:22.400 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:22.400 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:22.400 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:22.400 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:22.400 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:22.400 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:22.401 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:22.401 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:22.401 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:22.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.382+0000 7f933082c640 1 librados: starting msgr at 2024-07-26T23:35:22.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.382+0000 7f933082c640 1 librados: starting objecter 2024-07-26T23:35:22.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.382+0000 7f933082c640 1 Processor -- start 2024-07-26T23:35:22.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 -- start start 2024-07-26T23:35:22.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 librados: setting wanted keys 2024-07-26T23:35:22.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 librados: calling monclient init 2024-07-26T23:35:22.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient: init 2024-07-26T23:35:22.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:22.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:35:22.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c068a70 0x7f932c0ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:22.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): picked mon.noname-a con 0x7f932c068a70 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:22.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f932c0b0fc0 0x7f932c148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:22.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): picked mon.noname-c con 0x7f932c0b0fc0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:22.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c1411b0 0x7f932c14a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:22.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): picked mon.noname-b con 0x7f932c1411b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:22.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:22.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f932c141600 con 0x7f932c068a70 2024-07-26T23:35:22.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f932c147880 con 0x7f932c1411b0 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f932c0b0840 con 0x7f932c0b0fc0 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.383+0000 7f933082c640 10 monclient(hunting): authenticate will time out at 1355.419588s 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932a59c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c068a70 0x7f932c0ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f9321d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f932c0b0fc0 0x7f932c148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:22.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c1411b0 0x7f932c14a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932a59c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c068a70 0x7f932c0ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:60770/0 (socket says 172.21.15.139:60770) 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932a59c640 1 -- 172.21.15.139:0/3192546327 learned_addr learned my addr 172.21.15.139:0/3192546327 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932a59c640 10 monclient(hunting): get_auth_request con 0x7f932c068a70 auth_method 0 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932a59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932a59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f9321d9b640 10 monclient(hunting): get_auth_request con 0x7f932c0b0fc0 auth_method 0 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f9321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f9321d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f9321d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932ad9d640 10 monclient(hunting): get_auth_request con 0x7f932c1411b0 auth_method 0 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:22.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932ad9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.384+0000 7f932ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f932a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f932a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f932a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f932ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f932ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f932ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f9321d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f9321d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.385+0000 7f9321d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-07-26T23:35:22.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 1 -- 172.21.15.139:0/3192546327 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f932c0b0fc0 msgr2=0x7f932c148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f932c0b0fc0 0x7f932c148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 1 -- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c1411b0 msgr2=0x7f932c14a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c1411b0 0x7f932c14a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 1 monclient: found mon.noname-a 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.387+0000 7f932a59c640 1 -- 172.21.15.139:0/3192546327 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c149a10 con 0x7f932c068a70 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.390+0000 7f932a59c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c068a70 0x7f932c0ae8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f931c00c500 tx=0x7f931c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.390+0000 7f93237fe640 10 client.?.objecter ms_handle_connect 0x7f932c068a70 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.390+0000 7f93237fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 1 -- 172.21.15.139:0/3192546327 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0052d0 con 0x7f932c068a70 2024-07-26T23:35:22.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: dump: 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:22.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: _finish_auth 0 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: _check_auth_tickets 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 1 -- 172.21.15.139:0/3192546327 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f931c005470 con 0x7f932c068a70 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 1 -- 172.21.15.139:0/3192546327 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c012870 con 0x7f932c068a70 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f93237fe640 10 monclient: dump: 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:22.410 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f933082c640 5 monclient: authenticate success, global_id 4284 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f933082c640 10 monclient: _renew_subs 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f933082c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f932959a640 4 set_mon_vals no callback set 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.393+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932c149d50 con 0x7f932c068a70 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 10 monclient: _renew_subs 2024-07-26T23:35:22.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:22.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932c148d70 con 0x7f932c068a70 2024-07-26T23:35:22.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 1 librados: init done 2024-07-26T23:35:22.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:22.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:22.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:22.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.394+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f932c147170 con 0x7f932c068a70 2024-07-26T23:35:22.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.415+0000 7f93237fe640 1 -- 172.21.15.139:0/3192546327 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f931c012a10 con 0x7f932c068a70 2024-07-26T23:35:22.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.416+0000 7f93237fe640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f930c0789d0 0x7f930c07ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:22.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.416+0000 7f9321d9b640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f930c0789d0 0x7f930c07ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:22.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.416+0000 7f9321d9b640 10 monclient: get_auth_request con 0x7f930c0789d0 auth_method 0 2024-07-26T23:35:22.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.417+0000 7f93237fe640 10 client.4284.objecter ms_handle_connect 0x7f930c0789d0 2024-07-26T23:35:22.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.417+0000 7f9321d9b640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f930c0789d0 0x7f930c07ae90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f932c0afa60 tx=0x7f9314009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:22.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 1 -- 172.21.15.139:0/3192546327 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f931c012d00 con 0x7f932c068a70 2024-07-26T23:35:22.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 10 client.4284.objecter ms_dispatch 0x7f932c000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:22.656 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 3 client.4284.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:22.657 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 3 client.4284.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:22.657 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 20 client.4284.objecter dump_active .. 0 homeless 2024-07-26T23:35:22.657 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 1 -- 172.21.15.139:0/3192546327 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f932c149d50 con 0x7f932c068a70 2024-07-26T23:35:22.657 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:22.657 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.654+0000 7f93237fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:22.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.844+0000 7f933082c640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-07-26T23:35:22.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.844+0000 7f933082c640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-07-26T23:35:22.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.844+0000 7f933082c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:22.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.844+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f932c0b7fb0 con 0x7f932c068a70 2024-07-26T23:35:22.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.888+0000 7f93237fe640 1 -- 172.21.15.139:0/3192546327 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v6) ==== 86+0+158 (secure 0 0 0) 0x7f931c06a170 con 0x7f932c068a70 2024-07-26T23:35:22.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.888+0000 7f93237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-07-26T23:35:22.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.888+0000 7f93237fe640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:22.890 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.890+0000 7f933082c640 10 librados: watch_flush enter 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 10 librados: watch_flush exit 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 20 client.4284.objecter shutdown clearing up homeless session... 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 10 client.4284.objecter successfully canceled tick 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f930c0789d0 msgr2=0x7f930c07ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f930c0789d0 0x7f930c07ae90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f932c0afa60 tx=0x7f9314009040 comp rx=0 tx=0).stop 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 10 monclient: shutdown 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c068a70 msgr2=0x7f932c0ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c068a70 0x7f932c0ae8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f931c00c500 tx=0x7f931c00c9c0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 shutdown_connections 2024-07-26T23:35:22.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f930c0789d0 0x7f930c07ae90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f932c0b0fc0 0x7f932c148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c1411b0 0x7f932c14a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 --2- 172.21.15.139:0/3192546327 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f932c068a70 0x7f932c0ae8c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:22.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 >> 172.21.15.139:0/3192546327 conn(0x7f932c0b6670 msgr2=0x7f932c06e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:22.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 shutdown_connections 2024-07-26T23:35:22.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 -- 172.21.15.139:0/3192546327 wait complete. 2024-07-26T23:35:22.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:22.891+0000 7f933082c640 1 librados: shutdown 2024-07-26T23:35:22.903 INFO:teuthology.orchestra.run.smithi139.stdout:[{"version":6,"timestamp":"2024-07-26T23:35:12.695105+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"932.974381"}]}] 2024-07-26T23:35:22.904 INFO:tasks.ceph_manager:config epoch is 6 2024-07-26T23:35:22.904 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T23:35:22.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-07-26T23:35:22.904 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-07-26T23:35:23.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.006+0000 7fccc1849640 10 monclient: get_monmap_and_config 2024-07-26T23:35:23.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.006+0000 7fccc1849640 10 monclient: build_initial_monmap 2024-07-26T23:35:23.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.006+0000 7fccc1849640 10 monclient: monmap: 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:23.007280+0000 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:23.007280+0000 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:23.008 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.006+0000 7fccc1849640 1 Processor -- start 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.006+0000 7fccc1849640 1 -- start start 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.006+0000 7fccc1849640 10 monclient: init 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fccbc06e720 0x7fccbc139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient(hunting): picked mon.noname-c con 0x7fccbc06e720 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccbc143d00 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fccbc144cd0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:23.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccbc145810 con 0x7fccbc144cd0 2024-07-26T23:35:23.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccbc0ac810 con 0x7fccbc143d00 2024-07-26T23:35:23.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fccbc06eb00 con 0x7fccbc06e720 2024-07-26T23:35:23.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:23.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.007+0000 7fccc1849640 10 monclient(hunting): authenticate will time out at 1356.043505s 2024-07-26T23:35:23.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.014+0000 7fccbaffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fccbc06e720 0x7fccbc139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.014+0000 7fccbaffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fccbc06e720 0x7fccbc139bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41160/0 (socket says 172.21.15.139:41160) 2024-07-26T23:35:23.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.014+0000 7fccbaffd640 1 -- 172.21.15.139:0/4228216425 learned_addr learned my addr 172.21.15.139:0/4228216425 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:23.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.014+0000 7fccbaffd640 10 monclient(hunting): get_auth_request con 0x7fccbc06e720 auth_method 0 2024-07-26T23:35:23.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.014+0000 7fccbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.014+0000 7fccbaffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.014+0000 7fccbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.015+0000 7fccbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.015+0000 7fccbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.015+0000 7fccbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 10 monclient(hunting): handle_auth_done global_id 4276 payload 274 2024-07-26T23:35:23.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 1 -- 172.21.15.139:0/4228216425 >> v1:172.21.15.177:6789/0 conn(0x7fccbc143d00 legacy=0x7fccbc13a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 1 -- 172.21.15.139:0/4228216425 >> v1:172.21.15.139:6789/0 conn(0x7fccbc144cd0 legacy=0x7fccbc145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:23.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:23.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 1 monclient: found mon.noname-c 2024-07-26T23:35:23.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:23.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.017+0000 7fccbaffd640 1 -- 172.21.15.139:0/4228216425 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc0acaf0 con 0x7fccbc06e720 2024-07-26T23:35:23.022 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.021+0000 7fccbaffd640 1 --2- 172.21.15.139:0/4228216425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc06e720 0x7fccbc139bd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccac009870 tx=0x7fccac02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64b7e22b360d53d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 1 -- 172.21.15.139:0/4228216425 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac034070 con 0x7fccbc06e720 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: dump: 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.031 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 1 -- 172.21.15.139:0/4228216425 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccac030840 con 0x7fccbc06e720 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccc1849640 5 monclient: authenticate success, global_id 4276 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccc1849640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:23.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccc1849640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 1 -- 172.21.15.139:0/4228216425 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac030c10 con 0x7fccbc06e720 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccc1849640 4 set_mon_vals no callback set 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.029+0000 7fccb9ffb640 10 monclient: dump: 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.033 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.034 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.034 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.034 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 10 monclient: shutdown 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 1 -- 172.21.15.139:0/4228216425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc06e720 msgr2=0x7fccbc139bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 1 --2- 172.21.15.139:0/4228216425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc06e720 0x7fccbc139bd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccac009870 tx=0x7fccac02b980 comp rx=0 tx=0).stop 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 1 -- 172.21.15.139:0/4228216425 shutdown_connections 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 1 --2- 172.21.15.139:0/4228216425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc06e720 0x7fccbc139bd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 1 -- 172.21.15.139:0/4228216425 >> 172.21.15.139:0/4228216425 conn(0x7fccbc0b9c10 msgr2=0x7fccbc0ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 1 -- 172.21.15.139:0/4228216425 shutdown_connections 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.030+0000 7fccc1849640 1 -- 172.21.15.139:0/4228216425 wait complete. 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 10 monclient: build_initial_monmap 2024-07-26T23:35:23.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 10 monclient: monmap: 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 1 librados: starting msgr at 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 1 librados: starting objecter 2024-07-26T23:35:23.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 1 Processor -- start 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 1 -- start start 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 1 librados: setting wanted keys 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 1 librados: calling monclient init 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.031+0000 7fccc1849640 10 monclient: init 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fccbc06e720 0x7fccbc13c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): picked mon.noname-c con 0x7fccbc06e720 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc143d00 0x7fccbc1dc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): picked mon.noname-b con 0x7fccbc143d00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccbc144cd0 0x7fccbc134850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): picked mon.noname-a con 0x7fccbc144cd0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc141390 con 0x7fccbc144cd0 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc144710 con 0x7fccbc143d00 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fccbc14f170 con 0x7fccbc06e720 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccc1849640 10 monclient(hunting): authenticate will time out at 1356.068454s 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.032+0000 7fccba7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc143d00 0x7fccbc1dc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbb7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccbc144cd0 0x7fccbc134850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbaffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fccbc06e720 0x7fccbc13c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc143d00 0x7fccbc1dc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41168/0 (socket says 172.21.15.139:41168) 2024-07-26T23:35:23.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 1 -- 172.21.15.139:0/3888659101 learned_addr learned my addr 172.21.15.139:0/3888659101 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 10 monclient(hunting): get_auth_request con 0x7fccbc143d00 auth_method 0 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbb7fe640 10 monclient(hunting): get_auth_request con 0x7fccbc144cd0 auth_method 0 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbb7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbaffd640 10 monclient(hunting): get_auth_request con 0x7fccbc06e720 auth_method 0 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbaffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.033+0000 7fccba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.034+0000 7fccbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.034+0000 7fccbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.034+0000 7fccbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.034+0000 7fccbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.034+0000 7fccbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.034+0000 7fccbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 1 -- 172.21.15.139:0/3888659101 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fccbc06e720 msgr2=0x7fccbc13c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fccbc06e720 0x7fccbc13c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 1 -- 172.21.15.139:0/3888659101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccbc144cd0 msgr2=0x7fccbc134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccbc144cd0 0x7fccbc134850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 10 monclient: _finish_hunting 0 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 1 monclient: found mon.noname-b 2024-07-26T23:35:23.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.036+0000 7fccba7fc640 1 -- 172.21.15.139:0/3888659101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc13ea90 con 0x7fccbc143d00 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.038+0000 7fccba7fc640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc143d00 0x7fccbc1dc630 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccb000c530 tx=0x7fccb000c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.038+0000 7fcca3fff640 10 client.?.objecter ms_handle_connect 0x7fccbc143d00 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.038+0000 7fcca3fff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 1 -- 172.21.15.139:0/3888659101 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00052d0 con 0x7fccbc143d00 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: dump: 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.045 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:23.046 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 1 -- 172.21.15.139:0/3888659101 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb0005470 con 0x7fccbc143d00 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 1 -- 172.21.15.139:0/3888659101 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0012870 con 0x7fccbc143d00 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fcca3fff640 10 monclient: dump: 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.047 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 5 monclient: authenticate success, global_id 4279 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 10 monclient: _renew_subs 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccbc13ed00 con 0x7fccbc143d00 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccb97fa640 4 set_mon_vals no callback set 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 10 monclient: _renew_subs 2024-07-26T23:35:23.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccbc1392d0 con 0x7fccbc143d00 2024-07-26T23:35:23.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.042+0000 7fccc1849640 1 librados: init done 2024-07-26T23:35:23.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.043+0000 7fccc1849640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:23.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.043+0000 7fccc1849640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:23.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.043+0000 7fccc1849640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.043+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccbc139530 con 0x7fccbc143d00 2024-07-26T23:35:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.287+0000 7fcca3fff640 1 -- 172.21.15.139:0/3888659101 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fccb0012a10 con 0x7fccbc143d00 2024-07-26T23:35:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.287+0000 7fcca3fff640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fcca80789f0 0x7fcca807aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.288+0000 7fccbaffd640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fcca80789f0 0x7fcca807aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.288+0000 7fccbaffd640 10 monclient: get_auth_request con 0x7fcca80789f0 auth_method 0 2024-07-26T23:35:23.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.288+0000 7fccbaffd640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fcca80789f0 0x7fcca807aeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fccac036000 tx=0x7fccac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:23.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.288+0000 7fcca3fff640 10 client.4279.objecter ms_handle_connect 0x7fcca80789f0 2024-07-26T23:35:23.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.292+0000 7fcca3fff640 1 -- 172.21.15.139:0/3888659101 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fccb0012d00 con 0x7fccbc143d00 2024-07-26T23:35:23.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.292+0000 7fcca3fff640 10 client.4279.objecter ms_dispatch 0x7fccbc000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:23.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.292+0000 7fcca3fff640 3 client.4279.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:23.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.292+0000 7fcca3fff640 3 client.4279.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:23.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.292+0000 7fcca3fff640 20 client.4279.objecter dump_active .. 0 homeless 2024-07-26T23:35:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.304+0000 7fcca3fff640 1 -- 172.21.15.139:0/3888659101 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fccb0061900 con 0x7fccbc143d00 2024-07-26T23:35:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.304+0000 7fcca3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.304+0000 7fcca3fff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:23.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.541+0000 7fccc1849640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-07-26T23:35:23.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.541+0000 7fccc1849640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-07-26T23:35:23.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.541+0000 7fccc1849640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.541+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fccbc0ac250 con 0x7fccbc143d00 2024-07-26T23:35:23.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.675+0000 7fcca3fff640 1 -- 172.21.15.139:0/3888659101 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+191109 (secure 0 0 0) 0x7fccb006a190 con 0x7fccbc143d00 2024-07-26T23:35:23.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.675+0000 7fcca3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-07-26T23:35:23.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.675+0000 7fcca3fff640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:23.680 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.678+0000 7fccc1849640 10 librados: watch_flush enter 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.678+0000 7fccc1849640 10 librados: watch_flush exit 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.678+0000 7fccc1849640 20 client.4279.objecter shutdown clearing up homeless session... 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.678+0000 7fccc1849640 10 client.4279.objecter successfully canceled tick 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fcca80789f0 msgr2=0x7fcca807aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fcca80789f0 0x7fcca807aeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fccac036000 tx=0x7fccac008030 comp rx=0 tx=0).stop 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 10 monclient: shutdown 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc143d00 msgr2=0x7fccbc1dc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc143d00 0x7fccbc1dc630 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccb000c530 tx=0x7fccb000c9f0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 shutdown_connections 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fcca80789f0 0x7fcca807aeb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fccbc144cd0 0x7fccbc134850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc143d00 0x7fccbc1dc630 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 --2- 172.21.15.139:0/3888659101 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fccbc06e720 0x7fccbc13c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 >> 172.21.15.139:0/3888659101 conn(0x7fccbc0b9c10 msgr2=0x7fccbc14e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 shutdown_connections 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 -- 172.21.15.139:0/3888659101 wait complete. 2024-07-26T23:35:23.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.679+0000 7fccc1849640 1 librados: shutdown 2024-07-26T23:35:23.691 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1751160634},{"type":"v1","addr":"172.21.15.177:6801","nonce":1751160634}]},"active_addr":"172.21.15.177:6801/1751160634","active_change":"2024-07-26T23:34:19.551493+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":211721954}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3902250330}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":1496304332}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3427380456}]}]} 2024-07-26T23:35:23.697 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-07-26T23:35:23.697 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-07-26T23:35:23.697 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.832+0000 7fe168bd8640 10 monclient: get_monmap_and_config 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.832+0000 7fe168bd8640 10 monclient: build_initial_monmap 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.832+0000 7fe168bd8640 10 monclient: monmap: 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:23.833596+0000 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:23.833596+0000 2024-07-26T23:35:23.834 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:23.835 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.835 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:23.835 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:23.835 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:23.835 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:23.835 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:23.835 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 1 Processor -- start 2024-07-26T23:35:23.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 1 -- start start 2024-07-26T23:35:23.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient: init 2024-07-26T23:35:23.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:23.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-07-26T23:35:23.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1640b28d0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe1640749d0 0x7fe1640ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient(hunting): picked mon.noname-c con 0x7fe1640749d0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fe16413e8c0 0x7fe16413ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient(hunting): picked mon.noname-a con 0x7fe16413e8c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe16406f870 con 0x7fe1640b28d0 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fe1640b2cb0 con 0x7fe16413e8c0 2024-07-26T23:35:23.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe16406f620 con 0x7fe1640749d0 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.833+0000 7fe168bd8640 10 monclient(hunting): authenticate will time out at 1356.869981s 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.834+0000 7fe162d9d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fe16413e8c0 0x7fe16413ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.834+0000 7fe15bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe1640749d0 0x7fe1640ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe1640749d0 0x7fe1640ae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41178/0 (socket says 172.21.15.139:41178) 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 1 -- 172.21.15.139:0/2728813187 learned_addr learned my addr 172.21.15.139:0/2728813187 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 10 monclient(hunting): get_auth_request con 0x7fe1640749d0 auth_method 0 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe162d9d640 10 monclient(hunting): get_auth_request con 0x7fe16413e8c0 auth_method 0 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe162d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe162d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe162d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe15bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe162d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe162d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.835+0000 7fe162d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.841 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-07-26T23:35:23.841 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 1 -- 172.21.15.139:0/2728813187 >> v2:172.21.15.139:3300/0 conn(0x7fe16413e8c0 msgr2=0x7fe16413ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:23.841 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 1 --2- 172.21.15.139:0/2728813187 >> v2:172.21.15.139:3300/0 conn(0x7fe16413e8c0 0x7fe16413ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.841 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 1 -- 172.21.15.139:0/2728813187 >> v1:172.21.15.139:6789/0 conn(0x7fe1640b28d0 legacy=0x7fe164141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:23.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 1 monclient: found mon.noname-c 2024-07-26T23:35:23.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:23.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.837+0000 7fe15bfff640 1 -- 172.21.15.139:0/2728813187 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16406fb50 con 0x7fe1640749d0 2024-07-26T23:35:23.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.838+0000 7fe15bfff640 1 --2- 172.21.15.139:0/2728813187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe1640ae800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe16406ff00 tx=0x7fe14c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b83bf4d5efa11207 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 1 -- 172.21.15.139:0/2728813187 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe14c035020 con 0x7fe1640749d0 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: dump: 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.844 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: _finish_auth 0 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 1 -- 172.21.15.139:0/2728813187 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe14c031870 con 0x7fe1640749d0 2024-07-26T23:35:23.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 1 -- 172.21.15.139:0/2728813187 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe14c031c40 con 0x7fe1640749d0 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe168bd8640 5 monclient: authenticate success, global_id 4282 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe168bd8640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe168bd8640 4 set_mon_vals no callback set 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.842+0000 7fe161d9b640 10 monclient: dump: 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.846 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.847 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.847 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.847 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.847 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.847 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.847 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.847 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.851 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 10 monclient: shutdown 2024-07-26T23:35:23.851 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 -- 172.21.15.139:0/2728813187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 msgr2=0x7fe1640ae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 --2- 172.21.15.139:0/2728813187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe1640ae800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe16406ff00 tx=0x7fe14c02b910 comp rx=0 tx=0).stop 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 -- 172.21.15.139:0/2728813187 shutdown_connections 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 --2- 172.21.15.139:0/2728813187 >> v2:172.21.15.139:3300/0 conn(0x7fe16413e8c0 0x7fe16413ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 --2- 172.21.15.139:0/2728813187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe1640ae800 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 -- 172.21.15.139:0/2728813187 >> 172.21.15.139:0/2728813187 conn(0x7fe1640b5b40 msgr2=0x7fe1640b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 -- 172.21.15.139:0/2728813187 shutdown_connections 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.843+0000 7fe168bd8640 1 -- 172.21.15.139:0/2728813187 wait complete. 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 10 monclient: build_initial_monmap 2024-07-26T23:35:23.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 10 monclient: monmap: 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:23.853 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 1 librados: starting msgr at 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 1 librados: starting objecter 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 1 Processor -- start 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 1 -- start start 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 1 librados: setting wanted keys 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 1 librados: calling monclient init 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.844+0000 7fe168bd8640 10 monclient: init 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:35:23.854 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe164072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): picked mon.noname-b con 0x7fe1640749d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe1640b28d0 0x7fe16414a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): picked mon.noname-c con 0x7fe1640b28d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe16413e8c0 0x7fe16414abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): picked mon.noname-a con 0x7fe16413e8c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe16413f250 con 0x7fe16413e8c0 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe164141970 con 0x7fe1640749d0 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:23.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fe16414bfe0 con 0x7fe1640b28d0 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe168bd8640 10 monclient(hunting): authenticate will time out at 1356.881415s 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe15bfff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe1640b28d0 0x7fe16414a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.845+0000 7fe162d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe16413e8c0 0x7fe16414abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe164072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe16413e8c0 0x7fe16414abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:60790/0 (socket says 172.21.15.139:60790) 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 1 -- 172.21.15.139:0/1851087556 learned_addr learned my addr 172.21.15.139:0/1851087556 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 10 monclient(hunting): get_auth_request con 0x7fe1640749d0 auth_method 0 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 10 monclient(hunting): get_auth_request con 0x7fe16413e8c0 auth_method 0 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe15bfff640 10 monclient(hunting): get_auth_request con 0x7fe1640b28d0 auth_method 0 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe15bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe15bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe15bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe16259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe162d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe15bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe15bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.846+0000 7fe15bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-07-26T23:35:23.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 1 -- 172.21.15.139:0/1851087556 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe1640b28d0 msgr2=0x7fe16414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe1640b28d0 0x7fe16414a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 1 -- 172.21.15.139:0/1851087556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe16413e8c0 msgr2=0x7fe16414abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe16413e8c0 0x7fe16414abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 1 monclient: found mon.noname-b 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.849+0000 7fe16259c640 1 -- 172.21.15.139:0/1851087556 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe164146c40 con 0x7fe1640749d0 2024-07-26T23:35:23.861 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.851+0000 7fe16259c640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe164072bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe15400c3f0 tx=0x7fe15400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:23.861 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.851+0000 7fe15affd640 10 client.?.objecter ms_handle_connect 0x7fe1640749d0 2024-07-26T23:35:23.861 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.851+0000 7fe15affd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:23.861 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 1 -- 172.21.15.139:0/1851087556 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe154017070 con 0x7fe1640749d0 2024-07-26T23:35:23.861 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: dump: 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.862 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: _finish_auth 0 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 1 -- 172.21.15.139:0/1851087556 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1540052d0 con 0x7fe1640749d0 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 1 -- 172.21.15.139:0/1851087556 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe154003830 con 0x7fe1640749d0 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:23.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe15affd640 10 monclient: dump: 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:23.864 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 5 monclient: authenticate success, global_id 4285 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 monclient: _renew_subs 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe164146eb0 con 0x7fe1640749d0 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe16159a640 4 set_mon_vals no callback set 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 monclient: _renew_subs 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1640730f0 con 0x7fe1640749d0 2024-07-26T23:35:23.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 1 librados: init done 2024-07-26T23:35:23.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:23.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:23.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:23.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.856+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1640ae800 con 0x7fe1640749d0 2024-07-26T23:35:23.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.877+0000 7fe15affd640 1 -- 172.21.15.139:0/1851087556 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fe15401a070 con 0x7fe1640749d0 2024-07-26T23:35:23.879 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.877+0000 7fe15affd640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe1440789f0 0x7fe14407aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:23.879 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.878+0000 7fe15bfff640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe1440789f0 0x7fe14407aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:23.879 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.878+0000 7fe15bfff640 10 monclient: get_auth_request con 0x7fe1440789f0 auth_method 0 2024-07-26T23:35:23.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.878+0000 7fe15bfff640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe1440789f0 0x7fe14407aeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe14c036000 tx=0x7fe14c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:23.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.878+0000 7fe15affd640 10 client.4285.objecter ms_handle_connect 0x7fe1440789f0 2024-07-26T23:35:23.882 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.880+0000 7fe15affd640 1 -- 172.21.15.139:0/1851087556 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fe15405cd10 con 0x7fe1640749d0 2024-07-26T23:35:23.882 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.880+0000 7fe15affd640 10 client.4285.objecter ms_dispatch 0x7fe164000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:23.882 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.880+0000 7fe15affd640 3 client.4285.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:23.882 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.880+0000 7fe15affd640 3 client.4285.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:23.882 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:23.880+0000 7fe15affd640 20 client.4285.objecter dump_active .. 0 homeless 2024-07-26T23:35:24.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.120+0000 7fe15affd640 1 -- 172.21.15.139:0/1851087556 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fe154021020 con 0x7fe1640749d0 2024-07-26T23:35:24.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.120+0000 7fe15affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:24.121 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.120+0000 7fe15affd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:24.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.318+0000 7fe168bd8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:24.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.318+0000 7fe168bd8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:24.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.318+0000 7fe168bd8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:24.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.318+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe16413f670 con 0x7fe1640749d0 2024-07-26T23:35:24.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.373+0000 7fe15affd640 1 -- 172.21.15.139:0/1851087556 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+14885 (secure 0 0 0) 0x7fe154069310 con 0x7fe1640749d0 2024-07-26T23:35:24.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.373+0000 7fe15affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:24.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.373+0000 7fe15affd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:24.375 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:35:24.375 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":28,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:35:21.890950+0000","last_up_change":"2024-07-26T23:35:14.011640+0000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T23:35:00.571999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-07-26T23:35:15.932143+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6809","nonce":2123669081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6811","nonce":2123669081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6815","nonce":2123669081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6813","nonce":2123669081}]},"public_addr":"172.21.15.139:6809/2123669081","cluster_addr":"172.21.15.139:6811/2123669081","heartbeat_back_addr":"172.21.15.139:6815/2123669081","heartbeat_front_addr":"172.21.15.139:6813/2123669081","state":["exists","up"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6801","nonce":3107328002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6803","nonce":3107328002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6807","nonce":3107328002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6805","nonce":3107328002}]},"public_addr":"172.21.15.139:6801/3107328002","cluster_addr":"172.21.15.139:6803/3107328002","heartbeat_back_addr":"172.21.15.139:6807/3107328002","heartbeat_front_addr":"172.21.15.139:6805/3107328002","state":["exists","up"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6825","nonce":1043169956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6827","nonce":1043169956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6831","nonce":1043169956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6829","nonce":1043169956}]},"public_addr":"172.21.15.139:6825/1043169956","cluster_addr":"172.21.15.139:6827/1043169956","heartbeat_back_addr":"172.21.15.139:6831/1043169956","heartbeat_front_addr":"172.21.15.139:6829/1043169956","state":["exists","up"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6817","nonce":4033735752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6819","nonce":4033735752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6823","nonce":4033735752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6821","nonce":4033735752}]},"public_addr":"172.21.15.139:6817/4033735752","cluster_addr":"172.21.15.139:6819/4033735752","heartbeat_back_addr":"172.21.15.139:6823/4033735752","heartbeat_front_addr":"172.21.15.139:6821/4033735752","state":["exists","up"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6827","nonce":4112782318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6829","nonce":4112782318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6833","nonce":4112782318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6831","nonce":4112782318}]},"public_addr":"172.21.15.177:6827/4112782318","cluster_addr":"172.21.15.177:6829/4112782318","heartbeat_back_addr":"172.21.15.177:6833/4112782318","heartbeat_front_addr":"172.21.15.177:6831/4112782318","state":["exists","up"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6803","nonce":228413488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6805","nonce":228413488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6809","nonce":228413488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6807","nonce":228413488}]},"public_addr":"172.21.15.177:6803/228413488","cluster_addr":"172.21.15.177:6805/228413488","heartbeat_back_addr":"172.21.15.177:6809/228413488","heartbeat_front_addr":"172.21.15.177:6807/228413488","state":["exists","up"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6811","nonce":1761005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6813","nonce":1761005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6817","nonce":1761005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6815","nonce":1761005298}]},"public_addr":"172.21.15.177:6811/1761005298","cluster_addr":"172.21.15.177:6813/1761005298","heartbeat_back_addr":"172.21.15.177:6817/1761005298","heartbeat_front_addr":"172.21.15.177:6815/1761005298","state":["exists","up"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6819","nonce":3299104561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6821","nonce":3299104561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6825","nonce":3299104561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6823","nonce":3299104561}]},"public_addr":"172.21.15.177:6819/3299104561","cluster_addr":"172.21.15.177:6821/3299104561","heartbeat_back_addr":"172.21.15.177:6825/3299104561","heartbeat_front_addr":"172.21.15.177:6823/3299104561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:35:00.123097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.190400+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:35:00.557599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.842601+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:58.007938+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.331590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.423536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.831360+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.375+0000 7fe168bd8640 10 librados: watch_flush enter 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 10 librados: watch_flush exit 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 20 client.4285.objecter shutdown clearing up homeless session... 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 10 client.4285.objecter successfully canceled tick 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe1440789f0 msgr2=0x7fe14407aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe1440789f0 0x7fe14407aeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe14c036000 tx=0x7fe14c03c040 comp rx=0 tx=0).stop 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 10 monclient: shutdown 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 msgr2=0x7fe164072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe164072bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe15400c3f0 tx=0x7fe15400c8b0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 shutdown_connections 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe1440789f0 0x7fe14407aeb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe1640b28d0 0x7fe16414a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe16413e8c0 0x7fe16414abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 --2- 172.21.15.139:0/1851087556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1640749d0 0x7fe164072bb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 >> 172.21.15.139:0/1851087556 conn(0x7fe1640b5b40 msgr2=0x7fe16413fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 shutdown_connections 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 -- 172.21.15.139:0/1851087556 wait complete. 2024-07-26T23:35:24.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.376+0000 7fe168bd8640 1 librados: shutdown 2024-07-26T23:35:24.387 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-07-26T23:35:24.388 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T23:35:24.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.485+0000 7f1ec8b7f640 10 monclient: get_monmap_and_config 2024-07-26T23:35:24.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.485+0000 7f1ec8b7f640 10 monclient: build_initial_monmap 2024-07-26T23:35:24.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.485+0000 7f1ec8b7f640 10 monclient: monmap: 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:24.485249+0000 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:24.485249+0000 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:24.486 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:24.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.485+0000 7f1ec8b7f640 1 Processor -- start 2024-07-26T23:35:24.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.485+0000 7f1ec8b7f640 1 -- start start 2024-07-26T23:35:24.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.485+0000 7f1ec8b7f640 10 monclient: init 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec4140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec41470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ec4140e90 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ec40b48d0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec4147ce0 con 0x7f1ec40b48d0 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec40acaf0 con 0x7f1ec4140e90 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec41477c0 con 0x7f1ec41470c0 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.486+0000 7f1ec8b7f640 10 monclient(hunting): authenticate will time out at 1357.521684s 2024-07-26T23:35:24.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.487+0000 7f1ec259c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec4140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.487+0000 7f1ec2d9d640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f1ec40b48d0 0x7f1ec414e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:33264/0 (socket says 172.21.15.139:33264) 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.487+0000 7f1ec2d9d640 1 -- 172.21.15.139:0/2516325116 learned_addr learned my addr 172.21.15.139:0/2516325116 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.487+0000 7f1ec259c640 10 monclient(hunting): get_auth_request con 0x7f1ec41470c0 auth_method 0 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.487+0000 7f1ec259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.487+0000 7f1ec259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.487+0000 7f1ec259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.488+0000 7f1ec259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.488+0000 7f1ec259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:24.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.488+0000 7f1ec259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:24.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 1 -- 172.21.15.139:0/2516325116 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401171888 0 0) 0x7f1ec4147ce0 con 0x7f1ec40b48d0 2024-07-26T23:35:24.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 10 monclient(hunting): _negotiate 2024-07-26T23:35:24.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:24.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:24.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 10 monclient(hunting): my global_id is 4299 2024-07-26T23:35:24.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 1 -- 172.21.15.139:0/2516325116 --> v1:172.21.15.139:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb0003520 con 0x7f1ec40b48d0 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 1 -- 172.21.15.139:0/2516325116 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4584179 0 0) 0x7f1ec40acaf0 con 0x7f1ec4140e90 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 1 -- 172.21.15.139:0/2516325116 >> v1:172.21.15.177:6789/0 conn(0x7f1ec4140e90 legacy=0x7f1ec40b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 1 -- 172.21.15.139:0/2516325116 >> v1:172.21.15.139:6789/0 conn(0x7f1ec40b48d0 legacy=0x7f1ec414e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 1 monclient: found mon.noname-b 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec259c640 1 -- 172.21.15.139:0/2516325116 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec41474a0 con 0x7f1ec41470c0 2024-07-26T23:35:24.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.489+0000 7f1ec159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-07-26T23:35:24.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.492+0000 7f1ec259c640 1 --2- 172.21.15.139:0/2516325116 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec4140950 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009dc0 tx=0x7f1eb402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b69b65b04e1b243 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 1 -- 172.21.15.139:0/2516325116 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4039070 con 0x7f1ec41470c0 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: dump: 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.496 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: _finish_auth 0 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: _check_auth_tickets 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 1 -- 172.21.15.139:0/2516325116 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb402ccd0 con 0x7f1ec41470c0 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 1 -- 172.21.15.139:0/2516325116 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4034660 con 0x7f1ec41470c0 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec8b7f640 5 monclient: authenticate success, global_id 4250 2024-07-26T23:35:24.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec8b7f640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec8b7f640 4 set_mon_vals no callback set 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.495+0000 7f1ec159a640 10 monclient: dump: 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:24.498 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:24.499 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:24.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 10 monclient: shutdown 2024-07-26T23:35:24.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:24.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/2516325116 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 msgr2=0x7f1ec4140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/2516325116 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec4140950 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009dc0 tx=0x7f1eb402bf20 comp rx=0 tx=0).stop 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/2516325116 shutdown_connections 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/2516325116 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec4140950 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/2516325116 >> 172.21.15.139:0/2516325116 conn(0x7f1ec40b62c0 msgr2=0x7f1ec40b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/2516325116 shutdown_connections 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.496+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/2516325116 wait complete. 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 10 monclient: build_initial_monmap 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 10 monclient: monmap: 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:24.503 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 1 librados: starting msgr at 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 1 librados: starting objecter 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 1 Processor -- start 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 1 -- start start 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 1 librados: setting wanted keys 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 1 librados: calling monclient init 2024-07-26T23:35:24.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.497+0000 7f1ec8b7f640 10 monclient: init 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec40b48d0 0x7f1ec40ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec40b48d0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4140e90 0x7f1ec41d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec4140e90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec41d92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec41470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4141410 con 0x7f1ec40b48d0 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4074850 con 0x7f1ec4140e90 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:24.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec414f900 con 0x7f1ec41470c0 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.498+0000 7f1ec8b7f640 10 monclient(hunting): authenticate will time out at 1357.533547s 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec41d92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec40b48d0 0x7f1ec40ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec1d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4140e90 0x7f1ec41d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec41d92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60568/0 (socket says 172.21.15.139:60568) 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 1 -- 172.21.15.139:0/4045589778 learned_addr learned my addr 172.21.15.139:0/4045589778 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 10 monclient(hunting): get_auth_request con 0x7f1ec41470c0 auth_method 0 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec259c640 10 monclient(hunting): get_auth_request con 0x7f1ec40b48d0 auth_method 0 2024-07-26T23:35:24.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec1d9b640 10 monclient(hunting): get_auth_request con 0x7f1ec4140e90 auth_method 0 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec1d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.499+0000 7f1ec2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.500+0000 7f1ec259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.500+0000 7f1ec259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.500+0000 7f1ec259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.500+0000 7f1ec1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:24.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.500+0000 7f1ec1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:24.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.500+0000 7f1ec1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:24.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-07-26T23:35:24.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 1 -- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4140e90 msgr2=0x7f1ec41d8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:24.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4140e90 0x7f1ec41d8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 1 -- 172.21.15.139:0/4045589778 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec40b48d0 msgr2=0x7f1ec40ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:24.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec40b48d0 0x7f1ec40ad240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:24.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 10 monclient: _finish_hunting 0 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 1 monclient: found mon.noname-c 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.502+0000 7f1ec2d9d640 1 -- 172.21.15.139:0/4045589778 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec40af2a0 con 0x7f1ec41470c0 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.505+0000 7f1ec2d9d640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec41d92b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc00c460 tx=0x7f1ebc00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.505+0000 7f1ebaffd640 10 client.?.objecter ms_handle_connect 0x7f1ec41470c0 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.505+0000 7f1ebaffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:24.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 1 -- 172.21.15.139:0/4045589778 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc017070 con 0x7f1ec41470c0 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: dump: 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:24.511 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: _finish_auth 0 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 1 -- 172.21.15.139:0/4045589778 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ebc005ce0 con 0x7f1ec41470c0 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 1 -- 172.21.15.139:0/4045589778 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc007270 con 0x7f1ec41470c0 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ebaffd640 10 monclient: dump: 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.512 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 5 monclient: authenticate success, global_id 4253 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 10 monclient: _renew_subs 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec40af580 con 0x7f1ec41470c0 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec0d99640 4 set_mon_vals no callback set 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 10 monclient: _renew_subs 2024-07-26T23:35:24.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:24.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec40b2d30 con 0x7f1ec41470c0 2024-07-26T23:35:24.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.508+0000 7f1ec8b7f640 1 librados: init done 2024-07-26T23:35:24.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.509+0000 7f1ec8b7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:24.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.509+0000 7f1ec8b7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:24.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.509+0000 7f1ec8b7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:24.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.509+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec4140950 con 0x7f1ec41470c0 2024-07-26T23:35:24.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.529+0000 7f1ebaffd640 1 -- 172.21.15.139:0/4045589778 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f1ebc01a070 con 0x7f1ec41470c0 2024-07-26T23:35:24.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.529+0000 7f1ebaffd640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f1ea40789a0 0x7f1ea407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:24.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.530+0000 7f1ec259c640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f1ea40789a0 0x7f1ea407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:24.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.530+0000 7f1ec259c640 10 monclient: get_auth_request con 0x7f1ea40789a0 auth_method 0 2024-07-26T23:35:24.531 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.530+0000 7f1ec259c640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f1ea40789a0 0x7f1ea407ae60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4036000 tx=0x7f1eb4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:24.531 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.530+0000 7f1ebaffd640 10 client.4253.objecter ms_handle_connect 0x7f1ea40789a0 2024-07-26T23:35:24.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.533+0000 7f1ebaffd640 1 -- 172.21.15.139:0/4045589778 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f1ebc05cd70 con 0x7f1ec41470c0 2024-07-26T23:35:24.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.533+0000 7f1ebaffd640 10 client.4253.objecter ms_dispatch 0x7f1ec4000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:24.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.533+0000 7f1ebaffd640 3 client.4253.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:24.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.533+0000 7f1ebaffd640 3 client.4253.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:24.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.533+0000 7f1ebaffd640 20 client.4253.objecter dump_active .. 0 homeless 2024-07-26T23:35:24.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.771+0000 7f1ebaffd640 1 -- 172.21.15.139:0/4045589778 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f1ebc060ae0 con 0x7f1ec41470c0 2024-07-26T23:35:24.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.771+0000 7f1ebaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:24.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.771+0000 7f1ebaffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:24.955 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.955+0000 7f1ec8b7f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:24.956 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.955+0000 7f1ec8b7f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:24.956 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.955+0000 7f1ec8b7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:24.956 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:24.955+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ec40b8d30 con 0x7f1ec41470c0 2024-07-26T23:35:25.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.046+0000 7f1ebaffd640 1 -- 172.21.15.139:0/4045589778 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+14885 (secure 0 0 0) 0x7f1ebc065030 con 0x7f1ec41470c0 2024-07-26T23:35:25.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.046+0000 7f1ebaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:35:25.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.046+0000 7f1ebaffd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:25.047 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:35:25.047 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":28,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:35:21.890950+0000","last_up_change":"2024-07-26T23:35:14.011640+0000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T23:35:00.571999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-07-26T23:35:15.932143+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6809","nonce":2123669081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6811","nonce":2123669081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6815","nonce":2123669081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6813","nonce":2123669081}]},"public_addr":"172.21.15.139:6809/2123669081","cluster_addr":"172.21.15.139:6811/2123669081","heartbeat_back_addr":"172.21.15.139:6815/2123669081","heartbeat_front_addr":"172.21.15.139:6813/2123669081","state":["exists","up"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6801","nonce":3107328002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6803","nonce":3107328002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6807","nonce":3107328002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6805","nonce":3107328002}]},"public_addr":"172.21.15.139:6801/3107328002","cluster_addr":"172.21.15.139:6803/3107328002","heartbeat_back_addr":"172.21.15.139:6807/3107328002","heartbeat_front_addr":"172.21.15.139:6805/3107328002","state":["exists","up"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6825","nonce":1043169956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6827","nonce":1043169956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6831","nonce":1043169956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6829","nonce":1043169956}]},"public_addr":"172.21.15.139:6825/1043169956","cluster_addr":"172.21.15.139:6827/1043169956","heartbeat_back_addr":"172.21.15.139:6831/1043169956","heartbeat_front_addr":"172.21.15.139:6829/1043169956","state":["exists","up"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6817","nonce":4033735752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6819","nonce":4033735752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6823","nonce":4033735752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6821","nonce":4033735752}]},"public_addr":"172.21.15.139:6817/4033735752","cluster_addr":"172.21.15.139:6819/4033735752","heartbeat_back_addr":"172.21.15.139:6823/4033735752","heartbeat_front_addr":"172.21.15.139:6821/4033735752","state":["exists","up"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6827","nonce":4112782318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6829","nonce":4112782318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6833","nonce":4112782318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6831","nonce":4112782318}]},"public_addr":"172.21.15.177:6827/4112782318","cluster_addr":"172.21.15.177:6829/4112782318","heartbeat_back_addr":"172.21.15.177:6833/4112782318","heartbeat_front_addr":"172.21.15.177:6831/4112782318","state":["exists","up"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6803","nonce":228413488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6805","nonce":228413488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6809","nonce":228413488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6807","nonce":228413488}]},"public_addr":"172.21.15.177:6803/228413488","cluster_addr":"172.21.15.177:6805/228413488","heartbeat_back_addr":"172.21.15.177:6809/228413488","heartbeat_front_addr":"172.21.15.177:6807/228413488","state":["exists","up"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6811","nonce":1761005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6813","nonce":1761005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6817","nonce":1761005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6815","nonce":1761005298}]},"public_addr":"172.21.15.177:6811/1761005298","cluster_addr":"172.21.15.177:6813/1761005298","heartbeat_back_addr":"172.21.15.177:6817/1761005298","heartbeat_front_addr":"172.21.15.177:6815/1761005298","state":["exists","up"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6819","nonce":3299104561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6821","nonce":3299104561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6825","nonce":3299104561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6823","nonce":3299104561}]},"public_addr":"172.21.15.177:6819/3299104561","cluster_addr":"172.21.15.177:6821/3299104561","heartbeat_back_addr":"172.21.15.177:6825/3299104561","heartbeat_front_addr":"172.21.15.177:6823/3299104561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:35:00.123097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.190400+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:35:00.557599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.842601+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:58.007938+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.331590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.423536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.831360+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 10 librados: watch_flush enter 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 10 librados: watch_flush exit 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 20 client.4253.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 10 client.4253.objecter successfully canceled tick 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f1ea40789a0 msgr2=0x7f1ea407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f1ea40789a0 0x7f1ea407ae60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4036000 tx=0x7f1eb4030000 comp rx=0 tx=0).stop 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 10 monclient: shutdown 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 msgr2=0x7f1ec41d92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec41d92b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc00c460 tx=0x7f1ebc00c920 comp rx=0 tx=0).stop 2024-07-26T23:35:25.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 shutdown_connections 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f1ea40789a0 0x7f1ea407ae60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f1ec41470c0 0x7f1ec41d92b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4140e90 0x7f1ec41d8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 --2- 172.21.15.139:0/4045589778 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ec40b48d0 0x7f1ec40ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 >> 172.21.15.139:0/4045589778 conn(0x7f1ec40b62c0 msgr2=0x7f1ec40b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 shutdown_connections 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 -- 172.21.15.139:0/4045589778 wait complete. 2024-07-26T23:35:25.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.049+0000 7f1ec8b7f640 1 librados: shutdown 2024-07-26T23:35:25.062 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-07-26T23:35:25.062 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-07-26T23:35:25.063 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-07-26T23:35:25.063 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-07-26T23:35:25.063 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-07-26T23:35:25.064 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-07-26T23:35:25.064 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-07-26T23:35:25.065 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-07-26T23:35:25.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.232+0000 7f887c85a640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.232+0000 7f887c85a640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.232+0000 7f887c85a640 10 monclient: monmap: 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.232559+0000 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.232559+0000 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.234 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.233+0000 7f887c85a640 1 Processor -- start 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.233+0000 7f887c85a640 1 -- start start 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.233+0000 7f887c85a640 10 monclient: init 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-07-26T23:35:25.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f88780b10b0 0x7f88780aed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.236 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient(hunting): picked mon.noname-b con 0x7f88780b10b0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.236 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f887813c6f0 0x7f887813cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.236 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient(hunting): picked mon.noname-a con 0x7f887813c6f0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.236 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f887813f6a0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.236 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f887813d070 con 0x7f887813f6a0 2024-07-26T23:35:25.236 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.236 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f88780b1490 con 0x7f887813c6f0 2024-07-26T23:35:25.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f88780af840 con 0x7f88780b10b0 2024-07-26T23:35:25.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.234+0000 7f887c85a640 10 monclient(hunting): authenticate will time out at 1358.269896s 2024-07-26T23:35:25.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.236+0000 7f8876ffd640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f887813c6f0 0x7f887813cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.236+0000 7f8876ffd640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f887813c6f0 0x7f887813cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:60806/0 (socket says 172.21.15.139:60806) 2024-07-26T23:35:25.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.236+0000 7f8876ffd640 1 -- 172.21.15.139:0/904532222 learned_addr learned my addr 172.21.15.139:0/904532222 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.236+0000 7f8876ffd640 10 monclient(hunting): get_auth_request con 0x7f887813c6f0 auth_method 0 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.236+0000 7f8876ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.236+0000 7f8876ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.236+0000 7f8876ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7f88777fe640 1 --2- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f88780b10b0 0x7f88780aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7f88777fe640 10 monclient(hunting): get_auth_request con 0x7f88780b10b0 auth_method 0 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7f88777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7f88777fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7f88777fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.238+0000 7f8876ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.238+0000 7f8876ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.238+0000 7f8876ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.238+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.238+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.238+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.240 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7fa73a7a1640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.240 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7fa73a7a1640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.240 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.237+0000 7fa73a7a1640 10 monclient: monmap: 2024-07-26T23:35:25.240 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.237823+0000 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.237823+0000 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.241 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.242 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.242 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f993b71c640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f993b71c640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.242 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f993b71c640 10 monclient: monmap: 2024-07-26T23:35:25.242 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.243 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.243 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.240317+0000 2024-07-26T23:35:25.243 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.240317+0000 2024-07-26T23:35:25.243 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.243 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.243 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.241+0000 7f993b71c640 1 Processor -- start 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.241+0000 7f993b71c640 1 -- start start 2024-07-26T23:35:25.244 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.241+0000 7f993b71c640 10 monclient: init 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 1 -- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f88780b10b0 msgr2=0x7f88780aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 1 --2- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f88780b10b0 0x7f88780aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 1 -- 172.21.15.139:0/904532222 >> v1:172.21.15.177:6789/0 conn(0x7f887813f6a0 legacy=0x7f887813fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.245 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.246 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.240+0000 7f8876ffd640 1 -- 172.21.15.139:0/904532222 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88780ae320 con 0x7f887813c6f0 2024-07-26T23:35:25.246 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.246 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-07-26T23:35:25.246 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f99340b28d0 0x7f99341470d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.246 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient(hunting): picked mon.noname-c con 0x7f99340b28d0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.246 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934147610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.246 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient(hunting): picked mon.noname-b con 0x7f99340acda0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9934074260 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99340ae5d0 con 0x7f9934074260 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f99340b2cb0 con 0x7f99340b28d0 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f99340ad350 con 0x7f99340acda0 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.247 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f993b71c640 10 monclient(hunting): authenticate will time out at 1358.277601s 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f5582803640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f5582803640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7f5582803640 10 monclient: monmap: 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.242342+0000 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.242342+0000 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.248 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.249 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.249 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.249 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.249 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.249 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.250 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7fa73a7a1640 1 Processor -- start 2024-07-26T23:35:25.250 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7fa73a7a1640 1 -- start start 2024-07-26T23:35:25.250 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.242+0000 7fa73a7a1640 10 monclient: init 2024-07-26T23:35:25.250 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.251 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-07-26T23:35:25.251 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7340b1300 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.251 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa73413ca20 0x7fa734142d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.251 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient(hunting): picked mon.noname-c con 0x7fa73413ca20 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.251 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa73413ce30 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.251 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7f99394b8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f99340b28d0 0x7f99341470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.252 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7340ae5c0 con 0x7fa7340b1300 2024-07-26T23:35:25.252 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7340ae8a0 con 0x7fa73413ce30 2024-07-26T23:35:25.252 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7f5582803640 1 Processor -- start 2024-07-26T23:35:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa7340b16e0 con 0x7fa73413ca20 2024-07-26T23:35:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa73a7a1640 10 monclient(hunting): authenticate will time out at 1358.278443s 2024-07-26T23:35:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f5582803640 1 -- start start 2024-07-26T23:35:25.254 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7fa73979f640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7fa7340b1300 0x7fa7340aeda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:33280/0 (socket says 172.21.15.139:33280) 2024-07-26T23:35:25.254 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7fa73979f640 1 -- 172.21.15.139:0/1868053350 learned_addr learned my addr 172.21.15.139:0/1868053350 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:25.254 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f8876ffd640 1 --2- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f887813cb30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8870007980 tx=0x7f8870030490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdb2a273def34d4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.255 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f5582803640 10 monclient: init 2024-07-26T23:35:25.255 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f99394b8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f99340b28d0 0x7f99341470d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41210/0 (socket says 172.21.15.139:41210) 2024-07-26T23:35:25.255 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f99394b8640 1 -- 172.21.15.139:0/2286201055 learned_addr learned my addr 172.21.15.139:0/2286201055 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f99394b8640 10 monclient(hunting): get_auth_request con 0x7f99340b28d0 auth_method 0 2024-07-26T23:35:25.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f99394b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f99394b8640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7f99394b8640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1868053350 >> v2:172.21.15.177:3300/0 conn(0x7fa73413ca20 0x7fa734142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.257 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa3b5c29640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.257 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa3b5c29640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.257 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.243+0000 7fa3b5c29640 10 monclient: monmap: 2024-07-26T23:35:25.257 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.243168+0000 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.243168+0000 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.258 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.259 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7fa3b5c29640 1 Processor -- start 2024-07-26T23:35:25.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7fa3b5c29640 1 -- start start 2024-07-26T23:35:25.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.244+0000 7fa3b5c29640 10 monclient: init 2024-07-26T23:35:25.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f9938cb7640 1 --2- 172.21.15.139:0/2286201055 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934147610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f9938cb7640 10 monclient(hunting): get_auth_request con 0x7f99340acda0 auth_method 0 2024-07-26T23:35:25.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f9938cb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f9938cb7640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f9938cb7640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-07-26T23:35:25.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa3b00ace50 0x7fa3b00abde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): picked mon.noname-c con 0x7fa3b00ace50 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b00ad230 0x7fa3b0154270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-07-26T23:35:25.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f557c0b2730 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f557c0b10b0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f557c0aed00 0x7f557c13fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient(hunting): picked mon.noname-a con 0x7f557c0aed00 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557c0ae320 con 0x7f557c0b2730 2024-07-26T23:35:25.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557c0ae600 con 0x7f557c0b10b0 2024-07-26T23:35:25.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f557c0af840 con 0x7f557c0aed00 2024-07-26T23:35:25.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7f5582803640 10 monclient(hunting): authenticate will time out at 1358.280582s 2024-07-26T23:35:25.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): picked mon.noname-b con 0x7fa3b00ad230 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fa3b00ac320 0x7fa3b01547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): picked mon.noname-a con 0x7fa3b00ac320 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b0074c00 con 0x7fa3b00ac320 2024-07-26T23:35:25.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b00ae0a0 con 0x7fa3b00ace50 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b00ae5f0 con 0x7fa3b00ad230 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.245+0000 7fa3b5c29640 10 monclient(hunting): authenticate will time out at 1358.280966s 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.246+0000 7fa738f9e640 10 monclient(hunting): get_auth_request con 0x7fa73413ca20 auth_method 0 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.246+0000 7fa738f9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.246+0000 7fa738f9e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.246+0000 7fa738f9e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.246+0000 7f9938cb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.246+0000 7f9938cb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.265 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.246+0000 7f9938cb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.247+0000 7fa3aeffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b00ad230 0x7fa3b0154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.247+0000 7f5581801640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f557c0b2730 0x7f557c13f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:33284/0 (socket says 172.21.15.139:33284) 2024-07-26T23:35:25.266 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.247+0000 7f5581801640 1 -- 172.21.15.139:0/2834716921 learned_addr learned my addr 172.21.15.139:0/2834716921 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:25.267 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7f99394b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.267 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7f99394b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.267 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7f99394b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1806417228 0 0) 0x7fa7340ae8a0 con 0x7fa73413ce30 2024-07-26T23:35:25.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7fa72bfff640 10 monclient(hunting): _negotiate 2024-07-26T23:35:25.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7fa72bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7fa72bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7fa72bfff640 10 monclient(hunting): my global_id is 4297 2024-07-26T23:35:25.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa720003520 con 0x7fa73413ce30 2024-07-26T23:35:25.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7fa3affff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fa3b00ac320 0x7fa3b01547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7ff0ac1cf640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.248+0000 7ff0ac1cf640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7ff0ac1cf640 10 monclient: monmap: 2024-07-26T23:35:25.269 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.249071+0000 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.249071+0000 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.270 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 1 -- 172.21.15.139:0/2286201055 >> v2:172.21.15.177:3300/0 conn(0x7f99340b28d0 msgr2=0x7f99341470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 1 --2- 172.21.15.139:0/2286201055 >> v2:172.21.15.177:3300/0 conn(0x7f99340b28d0 0x7f99341470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 1 -- 172.21.15.139:0/2286201055 >> v1:172.21.15.139:6789/0 conn(0x7f9934074260 legacy=0x7f99340abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7f9938cb7640 1 -- 172.21.15.139:0/2286201055 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99340aea60 con 0x7f99340acda0 2024-07-26T23:35:25.272 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.249+0000 7ff0ac1cf640 1 Processor -- start 2024-07-26T23:35:25.272 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7ff0ac1cf640 1 -- start start 2024-07-26T23:35:25.272 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7ff0ac1cf640 10 monclient: init 2024-07-26T23:35:25.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7fa3af7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa3b00ace50 0x7fa3b00abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7fa3aeffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b00ad230 0x7fa3b0154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60590/0 (socket says 172.21.15.139:60590) 2024-07-26T23:35:25.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7fa3aeffd640 1 -- 172.21.15.139:0/250562286 learned_addr learned my addr 172.21.15.139:0/250562286 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7fa3aeffd640 10 monclient(hunting): get_auth_request con 0x7fa3b00ad230 auth_method 0 2024-07-26T23:35:25.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7fa3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7fa3aeffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7fa3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7ff0ac1cf640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.250+0000 7ff0ac1cf640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-07-26T23:35:25.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0a41470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7ff0a40b41c0 0x7ff0a40b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff0a40b41c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a414c1d0 0x7ff0a414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff0a414c1d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a4147ce0 con 0x7ff0a41470c0 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a40acaf0 con 0x7ff0a40b41c0 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a41477c0 con 0x7ff0a414c1d0 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7ff0ac1cf640 10 monclient(hunting): authenticate will time out at 1358.286305s 2024-07-26T23:35:25.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fa3affff640 10 monclient(hunting): get_auth_request con 0x7fa3b00ac320 auth_method 0 2024-07-26T23:35:25.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fa3affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fa3affff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fa3affff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fc428f9d640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fc428f9d640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.251+0000 7fc428f9d640 10 monclient: monmap: 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.251831+0000 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.251831+0000 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.277 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.278 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.278 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.278 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.278 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.278 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.278 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.278 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7fc428f9d640 1 Processor -- start 2024-07-26T23:35:25.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7f5582002640 1 --2- 172.21.15.139:0/2834716921 >> v2:172.21.15.139:3300/0 conn(0x7f557c0aed00 0x7f557c13fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7fc428f9d640 1 -- start start 2024-07-26T23:35:25.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7fc428f9d640 10 monclient: init 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7f5f81629640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7f5f81629640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7f5f81629640 10 monclient: monmap: 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.252866+0000 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.252866+0000 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.280 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.281 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.281 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.281 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.281 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.281 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7fa3af7fe640 10 monclient(hunting): get_auth_request con 0x7fa3b00ace50 auth_method 0 2024-07-26T23:35:25.281 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7fa3af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.281 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7fa3af7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.282 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.252+0000 7fa3af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-07-26T23:35:25.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fc42406e720 0x7fc424139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc42406e720 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424143d00 0x7fc42413a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc424143d00 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc424144cd0 0x7fc424145120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7f5f81629640 1 Processor -- start 2024-07-26T23:35:25.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc424144cd0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fc42406eb00 con 0x7fc42406e720 2024-07-26T23:35:25.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc424144710 con 0x7fc424144cd0 2024-07-26T23:35:25.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fc42413a650 con 0x7fc424143d00 2024-07-26T23:35:25.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fc428f9d640 10 monclient(hunting): authenticate will time out at 1358.288753s 2024-07-26T23:35:25.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7f5f81629640 1 -- start start 2024-07-26T23:35:25.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7f5f81629640 10 monclient: init 2024-07-26T23:35:25.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.253+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-07-26T23:35:25.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 1 -- 172.21.15.139:0/250562286 >> v2:172.21.15.177:3300/0 conn(0x7fa3b00ace50 msgr2=0x7fa3b00abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 1 --2- 172.21.15.139:0/250562286 >> v2:172.21.15.177:3300/0 conn(0x7fa3b00ace50 0x7fa3b00abde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 1 -- 172.21.15.139:0/250562286 >> v2:172.21.15.139:3300/0 conn(0x7fa3b00ac320 msgr2=0x7fa3b01547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 1 --2- 172.21.15.139:0/250562286 >> v2:172.21.15.139:3300/0 conn(0x7fa3b00ac320 0x7fa3b01547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7fa3aeffd640 1 -- 172.21.15.139:0/250562286 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b008dd30 con 0x7fa3b00ad230 2024-07-26T23:35:25.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-07-26T23:35:25.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f5f740387c0 0x7f5f74033370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient(hunting): picked mon.noname-a con 0x7f5f740387c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f74038ba0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 0x7f5f740cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient(hunting): picked mon.noname-b con 0x7f5f740338b0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f74036f70 con 0x7f5f74038ba0 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f5f740cb630 con 0x7f5f740387c0 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f5f74035500 con 0x7f5f740338b0 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.254+0000 7f5f81629640 10 monclient(hunting): authenticate will time out at 1358.289731s 2024-07-26T23:35:25.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.255+0000 7f5582002640 10 monclient(hunting): get_auth_request con 0x7f557c0aed00 auth_method 0 2024-07-26T23:35:25.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.255+0000 7f5582002640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.255+0000 7f5582002640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.255+0000 7f5582002640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.256+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.256+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.256+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.256+0000 7f5582002640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.256+0000 7f5582002640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.256+0000 7f5582002640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1634762318 0 0) 0x7fa720003520 con 0x7fa73413ce30 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 >> v2:172.21.15.177:3300/0 conn(0x7fa73413ca20 msgr2=0x7fa734142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 --2- 172.21.15.139:0/1868053350 >> v2:172.21.15.177:3300/0 conn(0x7fa73413ca20 0x7fa734142d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 >> v1:172.21.15.139:6789/0 conn(0x7fa7340b1300 legacy=0x7fa7340aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 monclient: found mon.noname-c-legacy 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa734143970 con 0x7fa73413ce30 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3006719369 0 0) 0x7fa7240016f0 con 0x7fa73413ce30 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa73a7a1640 5 monclient: authenticate success, global_id 4297 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa73a7a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: dump: 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.297 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 >> v1:172.21.15.177:6789/0 conn(0x7fa73413ce30 legacy=0x7fa734143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa720003aa0 0x7fa720003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): picked mon.a con 0x7fa720003aa0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa720004800 0x7fa720004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): picked mon.b con 0x7fa720004800 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa7200052e0 0x7fa72000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): picked mon.c con 0x7fa7200052e0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7340b16e0 con 0x7fa720003aa0 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa72000c1e0 con 0x7fa720004800 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa72000c3e0 con 0x7fa7200052e0 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa72bfff640 10 monclient(hunting): _finish_auth 0 2024-07-26T23:35:25.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7fa73a7a1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:25.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.257+0000 7f9938cb7640 1 --2- 172.21.15.139:0/2286201055 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934147610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99240097a0 tx=0x7f992402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f79ae645d8c957 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.258+0000 7fa3aeffd640 1 --2- 172.21.15.139:0/250562286 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b00ad230 0x7fa3b0154270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0069c30 tx=0x7fa39802de30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5c8480ea8c9e9f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7fc421d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424143d00 0x7fc42413a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-07-26T23:35:25.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 1 -- 172.21.15.139:0/2834716921 >> v1:172.21.15.177:6789/0 conn(0x7f557c0b10b0 legacy=0x7f557c147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 1 -- 172.21.15.139:0/2834716921 >> v1:172.21.15.139:6789/0 conn(0x7f557c0b2730 legacy=0x7f557c13f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.261+0000 7f5582002640 1 -- 172.21.15.139:0/2834716921 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c0ae8e0 con 0x7f557c0aed00 2024-07-26T23:35:25.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.262+0000 7ff0aa76c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a414c1d0 0x7ff0a414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.262+0000 7ff0a9f6b640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7ff0a41470c0 0x7ff0a4140950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:33286/0 (socket says 172.21.15.139:33286) 2024-07-26T23:35:25.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.262+0000 7ff0a9f6b640 1 -- 172.21.15.139:0/4061173337 learned_addr learned my addr 172.21.15.139:0/4061173337 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:25.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.263+0000 7fc422d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc424144cd0 0x7fc424145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.263+0000 7ff0aa76c640 10 monclient(hunting): get_auth_request con 0x7ff0a414c1d0 auth_method 0 2024-07-26T23:35:25.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.263+0000 7ff0aa76c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.263+0000 7ff0aa76c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.263+0000 7ff0aa76c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc422d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc424144cd0 0x7fc424145120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41240/0 (socket says 172.21.15.139:41240) 2024-07-26T23:35:25.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc422d9d640 1 -- 172.21.15.139:0/837972132 learned_addr learned my addr 172.21.15.139:0/837972132 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc422d9d640 10 monclient(hunting): get_auth_request con 0x7fc424144cd0 auth_method 0 2024-07-26T23:35:25.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc422d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc422d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc422d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc421d9b640 10 monclient(hunting): get_auth_request con 0x7fc424143d00 auth_method 0 2024-07-26T23:35:25.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc421d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc421d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.264+0000 7fc421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.265+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.265+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.265+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7f5582002640 1 --2- 172.21.15.139:0/2834716921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f557c0aed00 0x7f557c13fdb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec87ebb5ed0f2b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7fc422d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7fc422d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7fc422d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 1 -- 172.21.15.139:0/4061173337 >> v2:172.21.15.139:3300/0 conn(0x7ff0a40b41c0 msgr2=0x7ff0a40b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 1 --2- 172.21.15.139:0/4061173337 >> v2:172.21.15.139:3300/0 conn(0x7ff0a40b41c0 0x7ff0a40b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 1 -- 172.21.15.139:0/4061173337 >> v1:172.21.15.139:6789/0 conn(0x7ff0a41470c0 legacy=0x7ff0a4140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7ff0aa76c640 1 -- 172.21.15.139:0/4061173337 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a41474a0 con 0x7ff0a414c1d0 2024-07-26T23:35:25.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7f5f80e28640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 0x7f5f740cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7f5f7b7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f5f74038ba0 0x7f5f740ca9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:52016/0 (socket says 172.21.15.139:52016) 2024-07-26T23:35:25.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.266+0000 7f5f7b7fe640 1 -- 172.21.15.139:0/4285752862 learned_addr learned my addr 172.21.15.139:0/4285752862 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:25.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.267+0000 7fc421d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.267+0000 7fc421d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.267+0000 7fc421d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.267+0000 7f5f80e28640 10 monclient(hunting): get_auth_request con 0x7f5f740338b0 auth_method 0 2024-07-26T23:35:25.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.267+0000 7f5f80e28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.267+0000 7f5f80e28640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.267+0000 7f5f80e28640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.268+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa720004800 0x7fa720004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.268+0000 7f5f80e28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.268+0000 7f5f80e28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.269+0000 7f5f80e28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.269+0000 7fa738f9e640 10 monclient(hunting): get_auth_request con 0x7fa720004800 auth_method 0 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.269+0000 7fa738f9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.269+0000 7fa738f9e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.269+0000 7fa738f9e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7f5f7affd640 1 -- 172.21.15.139:0/4285752862 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2564210390 0 0) 0x7f5f74036f70 con 0x7f5f74038ba0 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7f5f7affd640 10 monclient(hunting): _negotiate 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7f5f7affd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7f5f7affd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7f5f7affd640 10 monclient(hunting): my global_id is 4315 2024-07-26T23:35:25.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7f5f7affd640 1 -- 172.21.15.139:0/4285752862 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f6c003520 con 0x7f5f74038ba0 2024-07-26T23:35:25.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.271+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-07-26T23:35:25.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-07-26T23:35:25.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 1 -- 172.21.15.139:0/837972132 >> v2:172.21.15.177:3300/0 conn(0x7fc424144cd0 msgr2=0x7fc424145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 1 --2- 172.21.15.139:0/837972132 >> v2:172.21.15.177:3300/0 conn(0x7fc424144cd0 0x7fc424145120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 1 -- 172.21.15.139:0/837972132 >> v2:172.21.15.139:3300/0 conn(0x7fc42406e720 msgr2=0x7fc424139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 1 --2- 172.21.15.139:0/837972132 >> v2:172.21.15.139:3300/0 conn(0x7fc42406e720 0x7fc424139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.272+0000 7fc421d9b640 1 -- 172.21.15.139:0/837972132 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4240ac810 con 0x7fc424143d00 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 1 -- 172.21.15.139:0/4285752862 >> v2:172.21.15.139:3300/0 conn(0x7f5f740387c0 msgr2=0x7f5f74033370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 1 --2- 172.21.15.139:0/4285752862 >> v2:172.21.15.139:3300/0 conn(0x7f5f740387c0 0x7f5f74033370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 1 -- 172.21.15.139:0/4285752862 >> v1:172.21.15.177:6789/0 conn(0x7f5f74038ba0 legacy=0x7f5f740ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7f5f80e28640 1 -- 172.21.15.139:0/4285752862 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f740395e0 con 0x7f5f740338b0 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 10 monclient(hunting): handle_auth_done global_id 4297 payload 293 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 1 -- 172.21.15.139:0/1868053350 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa7200052e0 msgr2=0x7fa72000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa7200052e0 0x7fa72000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 1 -- 172.21.15.139:0/1868053350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa720003aa0 msgr2=0x7fa720003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa720003aa0 0x7fa720003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 1 monclient: found mon.b 2024-07-26T23:35:25.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 1 -- 172.21.15.139:0/1868053350 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa720003520 con 0x7fa720004800 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa738f9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.273+0000 7fa739fa0640 1 -- 172.21.15.139:0/1868053350 reap_dead start 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.275+0000 7ff0aa76c640 1 --2- 172.21.15.139:0/4061173337 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a414c1d0 0x7ff0a414e5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009d90 tx=0x7ff09c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ec1a4eb7b5e8fc6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.276+0000 7f5f80e28640 1 --2- 172.21.15.139:0/4285752862 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 0x7f5f740cb0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c069dc0 tx=0x7f5f7c0a8120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b99c05da7e21323 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.277+0000 7fc421d9b640 1 --2- 172.21.15.139:0/837972132 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424143d00 0x7fc42413a110 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc42406f1f0 tx=0x7fc41802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba67a031e8f0b0d9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 1 -- 172.21.15.139:0/250562286 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398034070 con 0x7fa3b00ad230 2024-07-26T23:35:25.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: dump: 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.321 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 1 -- 172.21.15.139:0/250562286 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa39802ebe0 con 0x7fa3b00ad230 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 1 -- 172.21.15.139:0/250562286 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398004030 con 0x7fa3b00ad230 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3ae7fc640 10 monclient: dump: 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.322 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3b5c29640 5 monclient: authenticate success, global_id 4262 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3b5c29640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.280+0000 7fa3b5c29640 4 set_mon_vals no callback set 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 10 monclient: shutdown 2024-07-26T23:35:25.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 -- 172.21.15.139:0/250562286 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b00ad230 msgr2=0x7fa3b0154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 --2- 172.21.15.139:0/250562286 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b00ad230 0x7fa3b0154270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0069c30 tx=0x7fa39802de30 comp rx=0 tx=0).stop 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 -- 172.21.15.139:0/250562286 shutdown_connections 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 --2- 172.21.15.139:0/250562286 >> v2:172.21.15.139:3300/0 conn(0x7fa3b00ac320 0x7fa3b01547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 --2- 172.21.15.139:0/250562286 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b00ad230 0x7fa3b0154270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 --2- 172.21.15.139:0/250562286 >> v2:172.21.15.177:3300/0 conn(0x7fa3b00ace50 0x7fa3b00abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 -- 172.21.15.139:0/250562286 >> 172.21.15.139:0/250562286 conn(0x7fa3b00b5870 msgr2=0x7fa3b00b5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 -- 172.21.15.139:0/250562286 shutdown_connections 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.281+0000 7fa3b5c29640 1 -- 172.21.15.139:0/250562286 wait complete. 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.282+0000 7fa3b5c29640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.282+0000 7fa3b5c29640 10 monclient: monmap: 2024-07-26T23:35:25.324 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.282+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa720004800 0x7fa720004c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa73000a7f0 tx=0x7fa720005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a36900e9f7dbe4a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.283+0000 7fa3b5c29640 1 librados: starting msgr at 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.283+0000 7fa3b5c29640 1 librados: starting objecter 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.283+0000 7fa3b5c29640 1 Processor -- start 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.283+0000 7fa3b5c29640 1 -- start start 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.283+0000 7fa3b5c29640 1 librados: setting wanted keys 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.283+0000 7fa3b5c29640 1 librados: calling monclient init 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.283+0000 7fa3b5c29640 10 monclient: init 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:25.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3b00ac320 0x7fa3b0082470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): picked mon.noname-b con 0x7fa3b00ac320 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa3b00ace50 0x7fa3b01e9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): picked mon.noname-a con 0x7fa3b00ace50 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b01ea2f0 0x7fa3b008c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): picked mon.noname-c con 0x7fa3b01ea2f0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b00b1770 con 0x7fa3b00ace50 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b00b15f0 con 0x7fa3b00ac320 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b0154cf0 con 0x7fa3b01ea2f0 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.328 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.284+0000 7fa3b5c29640 10 monclient(hunting): authenticate will time out at 1358.319366s 2024-07-26T23:35:25.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73000b040 con 0x7fa720004800 2024-07-26T23:35:25.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 10 monclient: dump: 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa730037600 con 0x7fa720004800 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa73a7a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:25.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa73a7a1640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 1 -- 172.21.15.139:0/1868053350 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7300379d0 con 0x7fa720004800 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa73a7a1640 4 set_mon_vals no callback set 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa72bfff640 10 monclient: dump: 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.331 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.332 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.332 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.332 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.286+0000 7fa3af7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3b00ac320 0x7fa3b0082470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 1 -- 172.21.15.139:0/904532222 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887003e070 con 0x7f887813c6f0 2024-07-26T23:35:25.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: dump: 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.333 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 1 -- 172.21.15.139:0/904532222 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8870036480 con 0x7f887813c6f0 2024-07-26T23:35:25.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f887c85a640 5 monclient: authenticate success, global_id 4305 2024-07-26T23:35:25.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f887c85a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:25.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3aeffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa3b00ace50 0x7fa3b01e9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f887c85a640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f887c85a640 4 set_mon_vals no callback set 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 1 -- 172.21.15.139:0/2286201055 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9924031780 con 0x7f99340acda0 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: dump: 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.335 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 1 -- 172.21.15.139:0/904532222 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887000be60 con 0x7f887813c6f0 2024-07-26T23:35:25.336 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 1 -- 172.21.15.139:0/2286201055 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9924037070 con 0x7f99340acda0 2024-07-26T23:35:25.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 1 -- 172.21.15.139:0/2286201055 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9924031b40 con 0x7f99340acda0 2024-07-26T23:35:25.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f992bfff640 10 monclient: dump: 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.338 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f993b71c640 5 monclient: authenticate success, global_id 4259 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f993b71c640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f993b71c640 4 set_mon_vals no callback set 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7f88767fc640 10 monclient: dump: 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.339 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.340 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 10 monclient: shutdown 2024-07-26T23:35:25.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1868053350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa720004800 msgr2=0x7fa720004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa720004800 0x7fa720004c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa73000a7f0 tx=0x7fa720005830 comp rx=0 tx=0).stop 2024-07-26T23:35:25.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1868053350 shutdown_connections 2024-07-26T23:35:25.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 1 --2- 172.21.15.139:0/1868053350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa720004800 0x7fa720004c80 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa73000a7f0 tx=0x7fa720005830 comp rx=0 tx=0).stop 2024-07-26T23:35:25.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1868053350 >> 172.21.15.139:0/1868053350 conn(0x7fa7340b1c10 msgr2=0x7fa7340b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3aeffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa3b00ace50 0x7fa3b01e9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:60860/0 (socket says 172.21.15.139:60860) 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3aeffd640 1 -- 172.21.15.139:0/2859897221 learned_addr learned my addr 172.21.15.139:0/2859897221 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3aeffd640 10 monclient(hunting): get_auth_request con 0x7fa3b00ace50 auth_method 0 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3aeffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3af7fe640 10 monclient(hunting): get_auth_request con 0x7fa3b00ac320 auth_method 0 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3af7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa3af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1868053350 shutdown_connections 2024-07-26T23:35:25.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.287+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1868053350 wait complete. 2024-07-26T23:35:25.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 10 monclient: shutdown 2024-07-26T23:35:25.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 -- 172.21.15.139:0/2286201055 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 msgr2=0x7f9934147610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 --2- 172.21.15.139:0/2286201055 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934147610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99240097a0 tx=0x7f992402b820 comp rx=0 tx=0).stop 2024-07-26T23:35:25.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 10 monclient: shutdown 2024-07-26T23:35:25.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 -- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 msgr2=0x7f887813cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 --2- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f887813cb30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8870007980 tx=0x7f8870030490 comp rx=0 tx=0).stop 2024-07-26T23:35:25.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 -- 172.21.15.139:0/904532222 shutdown_connections 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 --2- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f887813cb30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 --2- 172.21.15.139:0/904532222 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f88780b10b0 0x7f88780aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 -- 172.21.15.139:0/904532222 >> 172.21.15.139:0/904532222 conn(0x7f88780b1c10 msgr2=0x7f88780b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 -- 172.21.15.139:0/2286201055 shutdown_connections 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 --2- 172.21.15.139:0/2286201055 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934147610 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 --2- 172.21.15.139:0/2286201055 >> v2:172.21.15.177:3300/0 conn(0x7f99340b28d0 0x7f99341470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 -- 172.21.15.139:0/2286201055 >> 172.21.15.139:0/2286201055 conn(0x7f99340b5b40 msgr2=0x7f99340b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.345 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 -- 172.21.15.139:0/904532222 shutdown_connections 2024-07-26T23:35:25.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f887c85a640 1 -- 172.21.15.139:0/904532222 wait complete. 2024-07-26T23:35:25.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 -- 172.21.15.139:0/2286201055 shutdown_connections 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7fa73a7a1640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7fa73a7a1640 10 monclient: monmap: 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.347 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.348 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.348 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.288+0000 7f993b71c640 1 -- 172.21.15.139:0/2286201055 wait complete. 2024-07-26T23:35:25.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa73a7a1640 1 librados: starting msgr at 2024-07-26T23:35:25.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa73a7a1640 1 librados: starting objecter 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f887c85a640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f887c85a640 10 monclient: monmap: 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.349 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 10 monclient: monmap: 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.350 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.351 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.351 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.351 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa73a7a1640 1 Processor -- start 2024-07-26T23:35:25.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa73a7a1640 1 -- start start 2024-07-26T23:35:25.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa73a7a1640 1 librados: setting wanted keys 2024-07-26T23:35:25.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa73a7a1640 1 librados: calling monclient init 2024-07-26T23:35:25.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa73a7a1640 10 monclient: init 2024-07-26T23:35:25.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 1 librados: starting msgr at 2024-07-26T23:35:25.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa3af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa3af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7fa3af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 1 librados: starting objecter 2024-07-26T23:35:25.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f887c85a640 1 librados: starting msgr at 2024-07-26T23:35:25.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f887c85a640 1 librados: starting objecter 2024-07-26T23:35:25.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 1 Processor -- start 2024-07-26T23:35:25.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f887c85a640 1 Processor -- start 2024-07-26T23:35:25.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 1 -- start start 2024-07-26T23:35:25.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 1 librados: setting wanted keys 2024-07-26T23:35:25.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 1 librados: calling monclient init 2024-07-26T23:35:25.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.289+0000 7f993b71c640 10 monclient: init 2024-07-26T23:35:25.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f887c85a640 1 -- start start 2024-07-26T23:35:25.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f887c85a640 1 librados: setting wanted keys 2024-07-26T23:35:25.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f887c85a640 1 librados: calling monclient init 2024-07-26T23:35:25.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f887c85a640 10 monclient: init 2024-07-26T23:35:25.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 1 -- 172.21.15.139:0/4285752862 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0b5070 con 0x7f5f740338b0 2024-07-26T23:35:25.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: dump: 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.357 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 1 -- 172.21.15.139:0/4285752862 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7c0b0440 con 0x7f5f740338b0 2024-07-26T23:35:25.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 1 -- 172.21.15.139:0/4285752862 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0b0870 con 0x7f5f740338b0 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f7affd640 10 monclient: dump: 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.359 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.360 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa73413dd40 0x7fa73413ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f81629640 5 monclient: authenticate success, global_id 4271 2024-07-26T23:35:25.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f81629640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f5f81629640 4 set_mon_vals no callback set 2024-07-26T23:35:25.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): picked mon.noname-b con 0x7fa73413dd40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa7341c9d50 0x7fa7341d84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): picked mon.noname-c con 0x7fa7341c9d50 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7341d8a10 0x7fa7341d8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:35:25.363 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): picked mon.noname-a con 0x7fa7341d8a10 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.363 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.363 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa7340aeda0 con 0x7fa7341d8a10 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7340b2940 con 0x7fa73413dd40 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa7340aef20 con 0x7fa7341c9d50 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7fa73a7a1640 10 monclient(hunting): authenticate will time out at 1358.325953s 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9934074260 0x7f99341492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): picked mon.noname-a con 0x7f9934074260 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.365 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.365 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.365 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:25.365 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88780b10b0 0x7f8878073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): picked mon.noname-c con 0x7f99340acda0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99340b28d0 0x7f9934145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): picked mon.noname-b con 0x7f99340b28d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f99340ac940 con 0x7f9934074260 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99340ac7c0 con 0x7f99340b28d0 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f99341525f0 con 0x7f99340acda0 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.290+0000 7f993b71c640 10 monclient(hunting): authenticate will time out at 1358.325793s 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): picked mon.noname-b con 0x7f88780b10b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f8878071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): picked mon.noname-a con 0x7f887813c6f0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f887813f6a0 0x7f8878071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): picked mon.noname-c con 0x7f887813f6a0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f88780b2730 con 0x7f887813c6f0 2024-07-26T23:35:25.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88780ae8e0 con 0x7f88780b10b0 2024-07-26T23:35:25.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f88780b28b0 con 0x7f887813f6a0 2024-07-26T23:35:25.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f887c85a640 10 monclient(hunting): authenticate will time out at 1358.326205s 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 10 monclient: shutdown 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 -- 172.21.15.139:0/4285752862 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 msgr2=0x7f5f740cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 --2- 172.21.15.139:0/4285752862 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 0x7f5f740cb0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c069dc0 tx=0x7f5f7c0a8120 comp rx=0 tx=0).stop 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 -- 172.21.15.139:0/4285752862 shutdown_connections 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 --2- 172.21.15.139:0/4285752862 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 0x7f5f740cb0f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 --2- 172.21.15.139:0/4285752862 >> v2:172.21.15.139:3300/0 conn(0x7f5f740387c0 0x7f5f74033370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 -- 172.21.15.139:0/4285752862 >> 172.21.15.139:0/4285752862 conn(0x7f5f74006580 msgr2=0x7f5f74006990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 -- 172.21.15.139:0/4285752862 shutdown_connections 2024-07-26T23:35:25.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.291+0000 7f5f81629640 1 -- 172.21.15.139:0/4285752862 wait complete. 2024-07-26T23:35:25.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-07-26T23:35:25.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 1 -- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b01ea2f0 msgr2=0x7fa3b008c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa3b01ea2f0 0x7fa3b008c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 1 -- 172.21.15.139:0/837972132 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc418031780 con 0x7fc424143d00 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: dump: 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.371 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 1 -- 172.21.15.139:0/837972132 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc418037070 con 0x7fc424143d00 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 1 -- 172.21.15.139:0/837972132 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc418031b40 con 0x7fc424143d00 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc42159a640 10 monclient: dump: 2024-07-26T23:35:25.372 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7f5f81629640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc428f9d640 5 monclient: authenticate success, global_id 4268 2024-07-26T23:35:25.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc428f9d640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fc428f9d640 4 set_mon_vals no callback set 2024-07-26T23:35:25.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7f5f81629640 10 monclient: monmap: 2024-07-26T23:35:25.374 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.374 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.375 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 1 -- 172.21.15.139:0/2859897221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3b00ac320 msgr2=0x7fa3b0082470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3b00ac320 0x7fa3b0082470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7fa3aeffd640 1 -- 172.21.15.139:0/2859897221 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b00845f0 con 0x7fa3b00ace50 2024-07-26T23:35:25.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7f5f81629640 1 librados: starting msgr at 2024-07-26T23:35:25.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.292+0000 7f5f81629640 1 librados: starting objecter 2024-07-26T23:35:25.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7f5f81629640 1 Processor -- start 2024-07-26T23:35:25.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7fc428f9d640 10 monclient: shutdown 2024-07-26T23:35:25.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7fc428f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7fc428f9d640 1 -- 172.21.15.139:0/837972132 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424143d00 msgr2=0x7fc42413a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7fc428f9d640 1 --2- 172.21.15.139:0/837972132 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424143d00 0x7fc42413a110 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc42406f1f0 tx=0x7fc41802b820 comp rx=0 tx=0).stop 2024-07-26T23:35:25.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7f5f81629640 1 -- start start 2024-07-26T23:35:25.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7f5f81629640 1 librados: setting wanted keys 2024-07-26T23:35:25.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7f5f81629640 1 librados: calling monclient init 2024-07-26T23:35:25.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7f5f81629640 10 monclient: init 2024-07-26T23:35:25.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7fc428f9d640 1 -- 172.21.15.139:0/837972132 shutdown_connections 2024-07-26T23:35:25.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7fc428f9d640 1 --2- 172.21.15.139:0/837972132 >> v2:172.21.15.177:3300/0 conn(0x7fc424144cd0 0x7fc424145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.293+0000 7fc428f9d640 1 --2- 172.21.15.139:0/837972132 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424143d00 0x7fc42413a110 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7fc428f9d640 1 --2- 172.21.15.139:0/837972132 >> v2:172.21.15.139:3300/0 conn(0x7fc42406e720 0x7fc424139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7fc428f9d640 1 -- 172.21.15.139:0/837972132 >> 172.21.15.139:0/837972132 conn(0x7fc4240b9c10 msgr2=0x7fc4240ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:35:25.381 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 0x7f5f740cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.381 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): picked mon.noname-c con 0x7f5f740338b0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.381 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f740387c0 0x7f5f740c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.381 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7fc428f9d640 1 -- 172.21.15.139:0/837972132 shutdown_connections 2024-07-26T23:35:25.381 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7fc428f9d640 1 -- 172.21.15.139:0/837972132 wait complete. 2024-07-26T23:35:25.382 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): picked mon.noname-a con 0x7f5f740387c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.382 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f740c7f00 0x7f5f740c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): picked mon.noname-b con 0x7f5f740c7f00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5f740d1d50 con 0x7f5f740387c0 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5f740352a0 con 0x7f5f740c7f00 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f5f740d1ed0 con 0x7f5f740338b0 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.383 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.294+0000 7f5f81629640 10 monclient(hunting): authenticate will time out at 1358.329341s 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 1 -- 172.21.15.139:0/4061173337 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c039070 con 0x7ff0a414c1d0 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: dump: 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.384 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 1 -- 172.21.15.139:0/4061173337 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09c02cbc0 con 0x7ff0a414c1d0 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 1 -- 172.21.15.139:0/4061173337 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c034440 con 0x7ff0a414c1d0 2024-07-26T23:35:25.385 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0a8f69640 10 monclient: dump: 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.386 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0ac1cf640 5 monclient: authenticate success, global_id 4265 2024-07-26T23:35:25.387 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0ac1cf640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.387 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7ff0ac1cf640 4 set_mon_vals no callback set 2024-07-26T23:35:25.387 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7fc428f9d640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.387 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7fc428f9d640 10 monclient: monmap: 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7fc428f9d640 1 librados: starting msgr at 2024-07-26T23:35:25.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.295+0000 7fc428f9d640 1 librados: starting objecter 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 1 Processor -- start 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 1 -- start start 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 1 librados: setting wanted keys 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 10 monclient: shutdown 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/4061173337 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a414c1d0 msgr2=0x7ff0a414e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 1 librados: calling monclient init 2024-07-26T23:35:25.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 10 monclient: init 2024-07-26T23:35:25.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 --2- 172.21.15.139:0/4061173337 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a414c1d0 0x7ff0a414e5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009d90 tx=0x7ff09c02be10 comp rx=0 tx=0).stop 2024-07-26T23:35:25.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/4061173337 shutdown_connections 2024-07-26T23:35:25.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 --2- 172.21.15.139:0/4061173337 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a414c1d0 0x7ff0a414e5c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 --2- 172.21.15.139:0/4061173337 >> v2:172.21.15.139:3300/0 conn(0x7ff0a40b41c0 0x7ff0a40b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/4061173337 >> 172.21.15.139:0/4061173337 conn(0x7ff0a40b62c0 msgr2=0x7ff0a40b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/4061173337 shutdown_connections 2024-07-26T23:35:25.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/4061173337 wait complete. 2024-07-26T23:35:25.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc42406e720 0x7fc4240b6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc42406e720 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc424143d00 0x7fc4240b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.296+0000 7fc428f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc424143d00 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424144cd0 0x7fc42413bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc424144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fc424141390 con 0x7fc424143d00 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4241441f0 con 0x7fc42406e720 2024-07-26T23:35:25.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fc424145660 con 0x7fc424144cd0 2024-07-26T23:35:25.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fc428f9d640 10 monclient(hunting): authenticate will time out at 1358.332106s 2024-07-26T23:35:25.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7ff0ac1cf640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7fa73979f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa73413dd40 0x7fa73413ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7ff0ac1cf640 10 monclient: monmap: 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.394 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.395 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.297+0000 7f9939cb9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99340b28d0 0x7f9934145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.395 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 librados: starting msgr at 2024-07-26T23:35:25.395 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 librados: starting objecter 2024-07-26T23:35:25.395 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 Processor -- start 2024-07-26T23:35:25.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa3aeffd640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa3b00ace50 0x7fa3b01e9db0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa39800bc20 tx=0x7fa3980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa3a7fff640 10 client.?.objecter ms_handle_connect 0x7fa3b00ace50 2024-07-26T23:35:25.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa3a7fff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 -- start start 2024-07-26T23:35:25.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 librados: setting wanted keys 2024-07-26T23:35:25.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 librados: calling monclient init 2024-07-26T23:35:25.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient: init 2024-07-26T23:35:25.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7f9939cb9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99340b28d0 0x7f9934145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41266/0 (socket says 172.21.15.139:41266) 2024-07-26T23:35:25.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7f9939cb9640 1 -- 172.21.15.139:0/975475855 learned_addr learned my addr 172.21.15.139:0/975475855 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7f9939cb9640 10 monclient(hunting): get_auth_request con 0x7f99340b28d0 auth_method 0 2024-07-26T23:35:25.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7f9939cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7f9939cb9640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7f9939cb9640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa73979f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa73413dd40 0x7fa73413ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41254/0 (socket says 172.21.15.139:41254) 2024-07-26T23:35:25.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa73979f640 1 -- 172.21.15.139:0/1917811410 learned_addr learned my addr 172.21.15.139:0/1917811410 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa73979f640 10 monclient(hunting): get_auth_request con 0x7fa73413dd40 auth_method 0 2024-07-26T23:35:25.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa73979f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa73979f640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7fa73979f640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:35:25.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0a40b41c0 0x7ff0a413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff0a40b41c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a41470c0 0x7ff0a4139e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): picked mon.noname-c con 0x7ff0a41470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7f5f80e28640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f740c7f00 0x7f5f740c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a414c1d0 0x7ff0a4138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff0a414c1d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a4141410 con 0x7ff0a40b41c0 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a4074850 con 0x7ff0a414c1d0 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a40b4b20 con 0x7ff0a41470c0 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.298+0000 7ff0ac1cf640 10 monclient(hunting): authenticate will time out at 1358.333967s 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f9939cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f9939cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f9939cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5f80e28640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f740c7f00 0x7f5f740c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41284/0 (socket says 172.21.15.139:41284) 2024-07-26T23:35:25.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5f80e28640 1 -- 172.21.15.139:0/2351394098 learned_addr learned my addr 172.21.15.139:0/2351394098 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5f80e28640 10 monclient(hunting): get_auth_request con 0x7f5f740c7f00 auth_method 0 2024-07-26T23:35:25.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5f80e28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5f80e28640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5f80e28640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 1 -- 172.21.15.139:0/2834716921 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c035020 con 0x7f557c0aed00 2024-07-26T23:35:25.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: dump: 2024-07-26T23:35:25.403 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.403 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 1 -- 172.21.15.139:0/2834716921 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c031830 con 0x7f557c0aed00 2024-07-26T23:35:25.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 1 -- 172.21.15.139:0/2834716921 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c031c00 con 0x7f557c0aed00 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5573fff640 10 monclient: dump: 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.405 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5582803640 5 monclient: authenticate success, global_id 4314 2024-07-26T23:35:25.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5582803640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.299+0000 7f5582803640 4 set_mon_vals no callback set 2024-07-26T23:35:25.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7fa73979f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7fa73979f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7fa73979f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 10 monclient: shutdown 2024-07-26T23:35:25.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 1 -- 172.21.15.139:0/2834716921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f557c0aed00 msgr2=0x7f557c13fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 1 --2- 172.21.15.139:0/2834716921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f557c0aed00 0x7f557c13fdb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02b8d0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 1 -- 172.21.15.139:0/2834716921 shutdown_connections 2024-07-26T23:35:25.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 1 --2- 172.21.15.139:0/2834716921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f557c0aed00 0x7f557c13fdb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 1 -- 172.21.15.139:0/2834716921 >> 172.21.15.139:0/2834716921 conn(0x7f557c0b1c10 msgr2=0x7f557c0b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 1 -- 172.21.15.139:0/2834716921 shutdown_connections 2024-07-26T23:35:25.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f5582803640 1 -- 172.21.15.139:0/2834716921 wait complete. 2024-07-26T23:35:25.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.300+0000 7f9938cb7640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5f7b7fe640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f740387c0 0x7f5f740c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 10 monclient: monmap: 2024-07-26T23:35:25.409 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 1 librados: starting msgr at 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 1 librados: starting objecter 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 1 Processor -- start 2024-07-26T23:35:25.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 1 -- start start 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 1 librados: setting wanted keys 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 1 librados: calling monclient init 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.301+0000 7f5582803640 10 monclient: init 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f557c0aed00 0x7f557c139b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-07-26T23:35:25.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): picked mon.noname-b con 0x7f557c0aed00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 1 -- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 msgr2=0x7f9934144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f99340acda0 0x7f9934144c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 1 -- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9934074260 msgr2=0x7f99341492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9934074260 0x7f99341492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f557c0b10b0 0x7f557c14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.413 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.413 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.413 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.413 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f9939cb9640 1 -- 172.21.15.139:0/975475855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993414ad70 con 0x7f99340b28d0 2024-07-26T23:35:25.413 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): picked mon.noname-a con 0x7f557c0b10b0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.414 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f557c0b2730 0x7f557c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.414 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): picked mon.noname-c con 0x7f557c0b2730 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.414 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.414 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa7341c9d50 0x7fa7341d84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f557c13cac0 con 0x7f557c0b10b0 2024-07-26T23:35:25.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f557c13c940 con 0x7f557c0aed00 2024-07-26T23:35:25.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f557c13cc40 con 0x7f557c0b2730 2024-07-26T23:35:25.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.302+0000 7f5582803640 10 monclient(hunting): authenticate will time out at 1358.337842s 2024-07-26T23:35:25.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.303+0000 7fa738f9e640 10 monclient(hunting): get_auth_request con 0x7fa7341c9d50 auth_method 0 2024-07-26T23:35:25.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.303+0000 7fa738f9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.303+0000 7fa738f9e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.303+0000 7fa738f9e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.303+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.303+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.303+0000 7fa738f9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.304+0000 7f5f80e28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.304+0000 7f5f80e28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.304+0000 7f5f80e28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.305+0000 7fa739fa0640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7341d8a10 0x7fa7341d8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.305+0000 7f5f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f740387c0 auth_method 0 2024-07-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.305+0000 7f5f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.305+0000 7f5f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.305+0000 7f5f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7f5f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7f5f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7f5f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-07-26T23:35:25.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 1 -- 172.21.15.139:0/1917811410 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa7341c9d50 msgr2=0x7fa7341d84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa7341c9d50 0x7fa7341d84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 1 -- 172.21.15.139:0/1917811410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7341d8a10 msgr2=0x7fa7341d8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7341d8a10 0x7fa7341d8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.306+0000 7fa73979f640 1 -- 172.21.15.139:0/1917811410 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73413f120 con 0x7fa73413dd40 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 1 -- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 msgr2=0x7f5f740cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5f740338b0 0x7f5f740cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 1 -- 172.21.15.139:0/2351394098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f740c7f00 msgr2=0x7f5f740c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f740c7f00 0x7f5f740c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.310+0000 7f5f7b7fe640 1 -- 172.21.15.139:0/2351394098 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f740cc2c0 con 0x7f5f740387c0 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.312+0000 7f8877fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f887813f6a0 0x7f8878071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.312+0000 7f9939cb9640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99340b28d0 0x7f9934145160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c730 tx=0x7f992c00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.422 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.312+0000 7f9929ffb640 10 client.?.objecter ms_handle_connect 0x7f99340b28d0 2024-07-26T23:35:25.422 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.312+0000 7f9929ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.422 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.312+0000 7fc421d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc424143d00 0x7fc4240b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.422 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.313+0000 7f8876ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f8878071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.423 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.313+0000 7fc422d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424144cd0 0x7fc42413bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.423 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7f8876ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f8878071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:60894/0 (socket says 172.21.15.139:60894) 2024-07-26T23:35:25.423 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7f8876ffd640 1 -- 172.21.15.139:0/3663648969 learned_addr learned my addr 172.21.15.139:0/3663648969 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.423 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7f8876ffd640 10 monclient(hunting): get_auth_request con 0x7f887813c6f0 auth_method 0 2024-07-26T23:35:25.423 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7f8876ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.423 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7f8876ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.423 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7f8876ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7fc421d9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc424143d00 0x7fc4240b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:60924/0 (socket says 172.21.15.139:60924) 2024-07-26T23:35:25.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7fc421d9b640 1 -- 172.21.15.139:0/4273215465 learned_addr learned my addr 172.21.15.139:0/4273215465 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7fc421d9b640 10 monclient(hunting): get_auth_request con 0x7fc424143d00 auth_method 0 2024-07-26T23:35:25.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7fc421d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7fc421d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7fc421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.314+0000 7ff0a976a640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a41470c0 0x7ff0a4139e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.427 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7f8877fff640 10 monclient(hunting): get_auth_request con 0x7f887813f6a0 auth_method 0 2024-07-26T23:35:25.427 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7f8877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.427 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7f8877fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.427 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7f8877fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7fc422d9d640 10 monclient(hunting): get_auth_request con 0x7fc424144cd0 auth_method 0 2024-07-26T23:35:25.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7fc422d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7fc422d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.315+0000 7fc422d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7fc421d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7fc421d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7fc421d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7ff0a976a640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a41470c0 0x7ff0a4139e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60688/0 (socket says 172.21.15.139:60688) 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7ff0a976a640 1 -- 172.21.15.139:0/2736713592 learned_addr learned my addr 172.21.15.139:0/2736713592 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7ff0a976a640 10 monclient(hunting): get_auth_request con 0x7ff0a41470c0 auth_method 0 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7ff0a976a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7ff0a976a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7ff0a976a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7f8876ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7f8876ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7f8876ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7fc422d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7fc422d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.316+0000 7fc422d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7fc42259c640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc42406e720 0x7fc4240b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7f5f7b7fe640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f740387c0 0x7f5f740c52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800c3c0 tx=0x7f5f6800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7f5f78ff9640 10 client.?.objecter ms_handle_connect 0x7f5f740387c0 2024-07-26T23:35:25.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7f5f78ff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0a976a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0a976a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0a976a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0aa76c640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a414c1d0 0x7ff0a4138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0aa76c640 10 monclient(hunting): get_auth_request con 0x7ff0a414c1d0 auth_method 0 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0aa76c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0aa76c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7ff0aa76c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7fa73979f640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa73413dd40 0x7fa73413ebe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa72c001a50 tx=0x7fa72c001f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7fa729ffb640 10 client.?.objecter ms_handle_connect 0x7fa73413dd40 2024-07-26T23:35:25.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.317+0000 7fa729ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.318+0000 7fc42259c640 10 monclient(hunting): get_auth_request con 0x7fc42406e720 auth_method 0 2024-07-26T23:35:25.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.318+0000 7fc42259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.318+0000 7fc42259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.318+0000 7fc42259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.319+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.319+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.319+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-07-26T23:35:25.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 1 -- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f887813f6a0 msgr2=0x7f8878071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f887813f6a0 0x7f8878071dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 1 -- 172.21.15.139:0/3663648969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88780b10b0 msgr2=0x7f8878073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88780b10b0 0x7f8878073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.320+0000 7f8876ffd640 1 -- 172.21.15.139:0/3663648969 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8878073160 con 0x7f887813c6f0 2024-07-26T23:35:25.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 1 -- 172.21.15.139:0/2736713592 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a41470c0 msgr2=0x7ff0a4139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff0a41470c0 0x7ff0a4139e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 1 -- 172.21.15.139:0/2736713592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0a40b41c0 msgr2=0x7ff0a413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0a40b41c0 0x7ff0a413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.322+0000 7ff0aa76c640 1 -- 172.21.15.139:0/2736713592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a4149200 con 0x7ff0a414c1d0 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 1 -- 172.21.15.139:0/4273215465 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424144cd0 msgr2=0x7fc42413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fc424144cd0 0x7fc42413bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 1 -- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc42406e720 msgr2=0x7fc4240b6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc42406e720 0x7fc4240b6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.323+0000 7fc421d9b640 1 -- 172.21.15.139:0/4273215465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4240b5860 con 0x7fc424143d00 2024-07-26T23:35:25.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f5581801640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f557c0aed00 0x7f557c139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c003b70 con 0x7f99340b28d0 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: dump: 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.439 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f992c003d30 con 0x7f99340b28d0 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c00aa90 con 0x7f99340b28d0 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f9929ffb640 10 monclient: dump: 2024-07-26T23:35:25.440 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 5 monclient: authenticate success, global_id 4321 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 10 monclient: _renew_subs 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993414b050 con 0x7f99340b28d0 2024-07-26T23:35:25.441 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f992b7fe640 4 set_mon_vals no callback set 2024-07-26T23:35:25.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7ff0aa76c640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a414c1d0 0x7ff0a4138b00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff09c033000 tx=0x7ff09c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 10 monclient: _renew_subs 2024-07-26T23:35:25.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7ff09a7fc640 10 client.?.objecter ms_handle_connect 0x7ff0a414c1d0 2024-07-26T23:35:25.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f993413db70 con 0x7f99340b28d0 2024-07-26T23:35:25.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7f993b71c640 1 librados: init done 2024-07-26T23:35:25.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.329+0000 7ff09a7fc640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.330+0000 7f993b71c640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.330+0000 7f993b71c640 15 client.4321.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.330+0000 7f993b71c640 20 client.4321.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.330+0000 7f993b71c640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.330+0000 7f993b71c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.330+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f99340abea0 con 0x7f99340b28d0 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f557c0aed00 0x7f557c139b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41308/0 (socket says 172.21.15.139:41308) 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 1 -- 172.21.15.139:0/4059357468 learned_addr learned my addr 172.21.15.139:0/4059357468 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 10 monclient(hunting): get_auth_request con 0x7f557c0aed00 auth_method 0 2024-07-26T23:35:25.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f8876ffd640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f8878071880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8870030a20 tx=0x7f8870004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f885ffff640 10 client.?.objecter ms_handle_connect 0x7f887813c6f0 2024-07-26T23:35:25.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f885ffff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.331+0000 7f5581801640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.332+0000 7f5582002640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f557c0b2730 0x7f557c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7f5582002640 10 monclient(hunting): get_auth_request con 0x7f557c0b2730 auth_method 0 2024-07-26T23:35:25.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7f5582002640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7f5582002640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7f5582002640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c044020 con 0x7ff0a414c1d0 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7ff09a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7ff09a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7ff09a7fc640 10 monclient: dump: 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.448 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7ff09a7fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.335+0000 7ff09a7fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09c0327f0 con 0x7ff0a414c1d0 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 5 monclient: authenticate success, global_id 4330 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 10 monclient: _renew_subs 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a413c280 con 0x7ff0a414c1d0 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09a7fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c018070 con 0x7fa73413dd40 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: dump: 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.450 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa72c00d6c0 con 0x7fa73413dd40 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c00db10 con 0x7fa73413dd40 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa729ffb640 10 monclient: dump: 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa72b7fe640 4 set_mon_vals no callback set 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09bfff640 4 set_mon_vals no callback set 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c039040 con 0x7ff0a414c1d0 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff09a7fc640 10 monclient: dump: 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.453 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 10 monclient: _renew_subs 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a413a540 con 0x7ff0a414c1d0 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7ff0ac1cf640 1 librados: init done 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 5 monclient: authenticate success, global_id 4324 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 10 monclient: _renew_subs 2024-07-26T23:35:25.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73413f900 con 0x7fa73413dd40 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 10 monclient: _renew_subs 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73413fec0 con 0x7fa73413dd40 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7fa73a7a1640 1 librados: init done 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7f5582002640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7f5582002640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.336+0000 7f5582002640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fc421d9b640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc424143d00 0x7fc4240b4360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc418009720 tx=0x7fc4180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fc412ffd640 10 client.?.objecter ms_handle_connect 0x7fc424143d00 2024-07-26T23:35:25.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fc412ffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7ff0ac1cf640 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7ff0ac1cf640 15 client.4330.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7ff0ac1cf640 20 client.4330.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7ff0ac1cf640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7ff0ac1cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0a41dbfb0 con 0x7ff0a414c1d0 2024-07-26T23:35:25.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fa73a7a1640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fa73a7a1640 15 client.4324.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fa73a7a1640 20 client.4324.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fa73a7a1640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fa73a7a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.458 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.337+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa734142d20 con 0x7fa73413dd40 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.338+0000 7f5581801640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.338+0000 7f5581801640 1 -- 172.21.15.139:0/4059357468 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f557c0b2730 msgr2=0x7f557c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f557c0b2730 0x7f557c0716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 1 -- 172.21.15.139:0/4059357468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f557c0b10b0 msgr2=0x7f557c14e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f557c0b10b0 0x7f557c14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.459 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5581801640 1 -- 172.21.15.139:0/4059357468 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c073170 con 0x7f557c0aed00 2024-07-26T23:35:25.460 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398003e50 con 0x7fa3b00ace50 2024-07-26T23:35:25.460 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.460 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.460 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: dump: 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.461 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa398045070 con 0x7fa3b00ace50 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398040410 con 0x7fa3b00ace50 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 5 monclient: authenticate success, global_id 4326 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 10 monclient: _renew_subs 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b00848d0 con 0x7fa3b00ace50 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3a7fff640 10 monclient: dump: 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.462 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3adffb640 4 set_mon_vals no callback set 2024-07-26T23:35:25.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 10 monclient: _renew_subs 2024-07-26T23:35:25.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b0082ee0 con 0x7fa3b00ace50 2024-07-26T23:35:25.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7fa3b5c29640 1 librados: init done 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68017070 con 0x7f5f740387c0 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: dump: 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.465 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f81629640 5 monclient: authenticate success, global_id 4329 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f68005ce0 con 0x7f5f740387c0 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f680072d0 con 0x7f5f740387c0 2024-07-26T23:35:25.466 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f78ff9640 10 monclient: dump: 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.467 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f81629640 10 monclient: _renew_subs 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f81629640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.339+0000 7f5f7a7fc640 4 set_mon_vals no callback set 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f740cd4d0 con 0x7f5f740387c0 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7f5f81629640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7f5f81629640 10 monclient: _renew_subs 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7f5f81629640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f740cda90 con 0x7f5f740387c0 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7f5f81629640 1 librados: init done 2024-07-26T23:35:25.468 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7fa3b5c29640 10 client.4326.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7fa3b5c29640 15 client.4326.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7fa3b5c29640 20 client.4326.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7fa3b5c29640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7fa3b5c29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.340+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa3b00abde0 con 0x7fa3b00ace50 2024-07-26T23:35:25.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.341+0000 7f5f81629640 10 client.4329.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.341+0000 7f5f81629640 15 client.4329.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.341+0000 7f5f81629640 20 client.4329.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.341+0000 7f5f81629640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.341+0000 7f5f81629640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.341+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5f7403aed0 con 0x7f5f740387c0 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887003e070 con 0x7f887813c6f0 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: dump: 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.471 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f887000ae50 con 0x7f887813c6f0 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870008040 con 0x7f887813c6f0 2024-07-26T23:35:25.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 5 monclient: authenticate success, global_id 4335 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 10 monclient: _renew_subs 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8878149a50 con 0x7f887813c6f0 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f885ffff640 10 monclient: dump: 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.473 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 10 monclient: _renew_subs 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f8875ffb640 4 set_mon_vals no callback set 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8878070d70 con 0x7f887813c6f0 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.356+0000 7f887c85a640 1 librados: init done 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.357+0000 7f887c85a640 10 client.4335.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.357+0000 7f887c85a640 15 client.4335.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.357+0000 7f887c85a640 20 client.4335.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.357+0000 7f887c85a640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.357+0000 7f887c85a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.357+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8878149870 con 0x7f887813c6f0 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc418032820 con 0x7fc424143d00 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: dump: 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.476 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4180329e0 con 0x7fc424143d00 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 5 monclient: authenticate success, global_id 4332 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 10 monclient: _renew_subs 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc418032dd0 con 0x7fc424143d00 2024-07-26T23:35:25.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 1 -- 172.21.15.139:0/4273215465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4240b5b40 con 0x7fc424143d00 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc420d99640 4 set_mon_vals no callback set 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc412ffd640 10 monclient: dump: 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.478 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 10 monclient: _renew_subs 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 1 -- 172.21.15.139:0/4273215465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4240b48a0 con 0x7fc424143d00 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.366+0000 7fc428f9d640 1 librados: init done 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.367+0000 7fc428f9d640 10 client.4332.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.367+0000 7fc428f9d640 15 client.4332.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.367+0000 7fc428f9d640 20 client.4332.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.367+0000 7fc428f9d640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.367+0000 7fc428f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.479 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.367+0000 7fc428f9d640 1 -- 172.21.15.139:0/4273215465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc42413d640 con 0x7fc424143d00 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.380+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f5f6801a070 con 0x7f5f740387c0 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.380+0000 7f5f78ff9640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5f600789f0 0x7f5f6007aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.381+0000 7f5f7bfff640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5f600789f0 0x7f5f6007aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.381+0000 7f5f7bfff640 10 monclient: get_auth_request con 0x7f5f600789f0 auth_method 0 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.381+0000 7f5f7bfff640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5f600789f0 0x7f5f6007aeb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009710 tx=0x7f5f70006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.381+0000 7f5f78ff9640 10 client.4329.objecter ms_handle_connect 0x7f5f600789f0 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f5f6805cc90 con 0x7f5f740387c0 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 10 client.4329.objecter ms_dispatch 0x7f5f7c000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 3 client.4329.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 3 client.4329.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 10 client.4329.objecter checking command 1 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f5f6007ee30 0x7f5f60081270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 20 client.4329.objecter _get_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 20 client.4329.objecter put_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 15 client.4329.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 20 client.4329.objecter _get_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 20 client.4329.objecter get_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 15 client.4329.objecter _session_command_op_assign 3 1 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 20 client.4329.objecter put_session s=0x7f5f6007e760 osd=3 4 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 10 client.4329.objecter _send_command 1 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f6801dec0 con 0x7f5f6007ee30 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.385+0000 7f5f78ff9640 20 client.4329.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.386+0000 7f5f80e28640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f5f6007ee30 0x7f5f60081270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.482 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.386+0000 7f5f80e28640 10 monclient: get_auth_request con 0x7f5f6007ee30 auth_method 0 2024-07-26T23:35:25.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.387+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f992c00ac30 con 0x7f99340b28d0 2024-07-26T23:35:25.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.387+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7ff09c047020 con 0x7ff0a414c1d0 2024-07-26T23:35:25.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.387+0000 7f9929ffb640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9918078a20 0x7f991807aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7ff09a7fc640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff0880789a0 0x7ff08807ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7f99394b8640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9918078a20 0x7f991807aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7f99394b8640 10 monclient: get_auth_request con 0x7f9918078a20 auth_method 0 2024-07-26T23:35:25.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7ff0a9f6b640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff0880789a0 0x7ff08807ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7ff0a9f6b640 10 monclient: get_auth_request con 0x7ff0880789a0 auth_method 0 2024-07-26T23:35:25.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7f9929ffb640 10 client.4321.objecter ms_handle_connect 0x7f9918078a20 2024-07-26T23:35:25.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7f99394b8640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9918078a20 0x7f991807aee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9930006f40 tx=0x7f9930006e90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7ff09a7fc640 10 client.4330.objecter ms_handle_connect 0x7ff0880789a0 2024-07-26T23:35:25.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.388+0000 7ff0a9f6b640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff0880789a0 0x7ff08807ae60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0005880 tx=0x7ff0a0007a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.389+0000 7f5f80e28640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f5f6007ee30 0x7f5f60081270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.389+0000 7f5f78ff9640 10 client.4329.objecter ms_handle_connect 0x7f5f6007ee30 2024-07-26T23:35:25.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f992c05ddb0 con 0x7f99340b28d0 2024-07-26T23:35:25.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 10 client.4321.objecter ms_dispatch 0x7f9934000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 3 client.4321.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 3 client.4321.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 10 client.4321.objecter checking command 1 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f991807ee60 0x7f99180812a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 20 client.4321.objecter _get_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 20 client.4321.objecter put_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 15 client.4321.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 20 client.4321.objecter _get_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 20 client.4321.objecter get_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 15 client.4321.objecter _session_command_op_assign 2 1 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 20 client.4321.objecter put_session s=0x7f991807e790 osd=2 4 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 10 client.4321.objecter _send_command 1 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 --> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9918081970 con 0x7f991807ee60 2024-07-26T23:35:25.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.395+0000 7f9929ffb640 20 client.4321.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.396+0000 7f9938cb7640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f991807ee60 0x7f99180812a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.396+0000 7f9938cb7640 10 monclient: get_auth_request con 0x7f991807ee60 auth_method 0 2024-07-26T23:35:25.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7ff09c000fb0 con 0x7ff0a414c1d0 2024-07-26T23:35:25.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 10 client.4330.objecter ms_dispatch 0x7ff0a4000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 3 client.4330.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 3 client.4330.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 10 client.4330.objecter checking command 1 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7ff08807ede0 0x7ff088081220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 20 client.4330.objecter _get_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 20 client.4330.objecter put_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 15 client.4330.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 20 client.4330.objecter _get_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 20 client.4330.objecter get_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 15 client.4330.objecter _session_command_op_assign 7 1 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 20 client.4330.objecter put_session s=0x7ff08807e710 osd=7 4 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 10 client.4330.objecter _send_command 1 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 --> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0880818f0 con 0x7ff08807ede0 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.399+0000 7ff09a7fc640 20 client.4330.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.400+0000 7ff0a976a640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7ff08807ede0 0x7ff088081220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.400+0000 7f5581801640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f557c0aed00 0x7f557c139b60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f557400cbc0 tx=0x7f557400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.400+0000 7f5571ffb640 10 client.?.objecter ms_handle_connect 0x7f557c0aed00 2024-07-26T23:35:25.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.400+0000 7f5571ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.400+0000 7ff0a976a640 10 monclient: get_auth_request con 0x7ff08807ede0 auth_method 0 2024-07-26T23:35:25.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.401+0000 7f9938cb7640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f991807ee60 0x7f99180812a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.401+0000 7f9929ffb640 10 client.4321.objecter ms_handle_connect 0x7f991807ee60 2024-07-26T23:35:25.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.404+0000 7ff09a7fc640 10 client.4330.objecter ms_handle_connect 0x7ff08807ede0 2024-07-26T23:35:25.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.404+0000 7ff0a976a640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7ff08807ede0 0x7ff088081220 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.428+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fa72c01b070 con 0x7fa73413dd40 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.429+0000 7fa729ffb640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa7180789f0 0x7fa71807aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.429+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa7180789f0 0x7fa71807aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.429+0000 7fa738f9e640 10 monclient: get_auth_request con 0x7fa7180789f0 auth_method 0 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.430+0000 7fa729ffb640 10 client.4324.objecter ms_handle_connect 0x7fa7180789f0 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.430+0000 7fa738f9e640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa7180789f0 0x7fa71807aeb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa73003a000 tx=0x7fa730035000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fa72c05d550 con 0x7fa73413dd40 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 10 client.4324.objecter ms_dispatch 0x7fa734000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 3 client.4324.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 10 client.4324.objecter checking command 1 2024-07-26T23:35:25.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7fa71807ee30 0x7fa718081270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 20 client.4324.objecter _get_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 20 client.4324.objecter put_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 15 client.4324.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 20 client.4324.objecter _get_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 20 client.4324.objecter get_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 15 client.4324.objecter _session_command_op_assign 5 1 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 20 client.4324.objecter put_session s=0x7fa71807e760 osd=5 4 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 10 client.4324.objecter _send_command 1 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 --> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa72c00de50 con 0x7fa71807ee30 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.432+0000 7fa729ffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.433+0000 7fa739fa0640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7fa71807ee30 0x7fa718081270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.433+0000 7fa739fa0640 10 monclient: get_auth_request con 0x7fa71807ee30 auth_method 0 2024-07-26T23:35:25.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574017070 con 0x7f557c0aed00 2024-07-26T23:35:25.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5571ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5571ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5571ffb640 10 monclient: dump: 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5571ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5571ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5571ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5582803640 5 monclient: authenticate success, global_id 4336 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5582803640 10 monclient: _renew_subs 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5582803640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.436+0000 7f5582803640 1 -- 172.21.15.139:0/4059357468 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557c073450 con 0x7f557c0aed00 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.437+0000 7fa739fa0640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7fa71807ee30 0x7fa718081270 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.437+0000 7fa729ffb640 10 client.4324.objecter ms_handle_connect 0x7fa71807ee30 2024-07-26T23:35:25.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.437+0000 7f5582803640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.437+0000 7f5582803640 10 monclient: _renew_subs 2024-07-26T23:35:25.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.437+0000 7f5582803640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.437+0000 7f5582803640 1 -- 172.21.15.139:0/4059357468 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557c073b50 con 0x7f557c0aed00 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.437+0000 7f5582803640 1 librados: init done 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.438+0000 7f5582803640 10 client.4336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.438+0000 7f5582803640 15 client.4336.objecter _session_command_op_assign -1 1 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.438+0000 7f5582803640 20 client.4336.objecter _get_session osd=-1 returning homeless 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.438+0000 7f5582803640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.438+0000 7f5582803640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.438+0000 7f5582803640 1 -- 172.21.15.139:0/4059357468 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f557c073e00 con 0x7f557c0aed00 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.462+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 <== osd.3 v2:172.21.15.139:6816/4033735752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f6801dec0 con 0x7f5f6007ee30 2024-07-26T23:35:25.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.462+0000 7f5f78ff9640 10 client.4329.objecter ms_dispatch 0x7f5f7c000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.462+0000 7f5f78ff9640 10 client.4329.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.462+0000 7f5f78ff9640 20 client.4329.objecter put_session s=0x7f5f6007e760 osd=3 4 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.462+0000 7f5f78ff9640 15 client.4329.objecter _session_command_op_remove 3 1 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.465+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 <== osd.7 v2:172.21.15.177:6818/3299104561 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0880818f0 con 0x7ff08807ede0 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.465+0000 7ff09a7fc640 10 client.4330.objecter ms_dispatch 0x7ff0a4000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.465+0000 7ff09a7fc640 10 client.4330.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.465+0000 7ff09a7fc640 20 client.4330.objecter put_session s=0x7ff08807e710 osd=7 4 2024-07-26T23:35:25.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.465+0000 7ff09a7fc640 15 client.4330.objecter _session_command_op_remove 7 1 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 10 client.4329.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 15 client.4329.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 20 client.4329.objecter _get_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 20 client.4329.objecter put_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 20 client.4329.objecter _get_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 15 client.4329.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 20 client.4329.objecter get_session s=0x7f5f6007e760 osd=3 3 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 15 client.4329.objecter _session_command_op_assign 3 2 2024-07-26T23:35:25.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 20 client.4329.objecter put_session s=0x7f5f6007e760 osd=3 4 2024-07-26T23:35:25.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 10 client.4329.objecter _send_command 2 2024-07-26T23:35:25.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.476+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5f74032540 con 0x7f5f6007ee30 2024-07-26T23:35:25.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.481+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 <== osd.5 v2:172.21.15.177:6802/228413488 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa72c00de50 con 0x7fa71807ee30 2024-07-26T23:35:25.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.481+0000 7fa729ffb640 10 client.4324.objecter ms_dispatch 0x7fa734000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.481+0000 7fa729ffb640 10 client.4324.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.481+0000 7fa729ffb640 20 client.4324.objecter put_session s=0x7fa71807e760 osd=5 4 2024-07-26T23:35:25.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.481+0000 7fa729ffb640 15 client.4324.objecter _session_command_op_remove 5 1 2024-07-26T23:35:25.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f5f78ff9640 1 -- 172.21.15.139:0/2351394098 <== osd.3 v2:172.21.15.139:6816/4033735752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5f74032540 con 0x7f5f6007ee30 2024-07-26T23:35:25.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f5f78ff9640 10 client.4329.objecter ms_dispatch 0x7f5f7c000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f5f78ff9640 10 client.4329.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f5f78ff9640 20 client.4329.objecter put_session s=0x7f5f6007e760 osd=3 4 2024-07-26T23:35:25.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f5f78ff9640 15 client.4329.objecter _session_command_op_remove 3 2 2024-07-26T23:35:25.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 <== osd.2 v2:172.21.15.139:6824/1043169956 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9918081970 con 0x7f991807ee60 2024-07-26T23:35:25.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f9929ffb640 10 client.4321.objecter ms_dispatch 0x7f9934000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f9929ffb640 10 client.4321.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f9929ffb640 20 client.4321.objecter put_session s=0x7f991807e790 osd=2 4 2024-07-26T23:35:25.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.488+0000 7f9929ffb640 15 client.4321.objecter _session_command_op_remove 2 1 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 10 librados: watch_flush enter 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 10 librados: watch_flush exit 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 10 client.4329.objecter close_session for osd.3 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f5f6007ee30 msgr2=0x7f5f60081270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f5f6007ee30 0x7f5f60081270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 20 client.4329.objecter put_session s=0x7f5f6007e760 osd=3 1 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 20 client.4329.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 10 client.4329.objecter successfully canceled tick 2024-07-26T23:35:25.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5f600789f0 msgr2=0x7f5f6007aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5f600789f0 0x7f5f6007aeb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009710 tx=0x7f5f70006c90 comp rx=0 tx=0).stop 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 10 monclient: shutdown 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 20 monclient: shutdown canceling and discarding version request 1 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f740387c0 msgr2=0x7f5f740c52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 --2- 172.21.15.139:0/2351394098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f740387c0 0x7f5f740c52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800c3c0 tx=0x7f5f6800c880 comp rx=0 tx=0).stop 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f80e28640 1 -- 172.21.15.139:0/2351394098 reap_dead start 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 shutdown_connections 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.489+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 >> 172.21.15.139:0/2351394098 conn(0x7f5f74006580 msgr2=0x7f5f74037f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.490+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 shutdown_connections 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.490+0000 7f5f81629640 1 -- 172.21.15.139:0/2351394098 wait complete. 2024-07-26T23:35:25.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.490+0000 7f5f81629640 1 librados: shutdown 2024-07-26T23:35:25.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 15 client.4330.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 20 client.4330.objecter _get_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 20 client.4330.objecter put_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 20 client.4330.objecter _get_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 15 client.4330.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.507 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 20 client.4330.objecter get_session s=0x7ff08807e710 osd=7 3 2024-07-26T23:35:25.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 15 client.4330.objecter _session_command_op_assign 7 2 2024-07-26T23:35:25.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 20 client.4330.objecter put_session s=0x7ff08807e710 osd=7 4 2024-07-26T23:35:25.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 10 client.4330.objecter _send_command 2 2024-07-26T23:35:25.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.491+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 --> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff0a41dbf50 con 0x7ff08807ede0 2024-07-26T23:35:25.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.495+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f8870005d10 con 0x7f887813c6f0 2024-07-26T23:35:25.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.495+0000 7f885ffff640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f88640789a0 0x7f886407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.495+0000 7f88777fe640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f88640789a0 0x7f886407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.495+0000 7f88777fe640 10 monclient: get_auth_request con 0x7f88640789a0 auth_method 0 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.496+0000 7f88777fe640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f88640789a0 0x7f886407ae60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8868006a90 tx=0x7f8868006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.496+0000 7f885ffff640 10 client.4335.objecter ms_handle_connect 0x7f88640789a0 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff09a7fc640 1 -- 172.21.15.139:0/2736713592 <== osd.7 v2:172.21.15.177:6818/3299104561 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff0a41dbf50 con 0x7ff08807ede0 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff09a7fc640 10 client.4330.objecter ms_dispatch 0x7ff0a4000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff09a7fc640 10 client.4330.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff09a7fc640 20 client.4330.objecter put_session s=0x7ff08807e710 osd=7 4 2024-07-26T23:35:25.510 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff09a7fc640 15 client.4330.objecter _session_command_op_remove 7 2 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 10 librados: watch_flush enter 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 10 librados: watch_flush exit 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 10 client.4330.objecter close_session for osd.7 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7ff08807ede0 msgr2=0x7ff088081220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7ff08807ede0 0x7ff088081220 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 20 client.4330.objecter put_session s=0x7ff08807e710 osd=7 1 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 20 client.4330.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 10 client.4330.objecter successfully canceled tick 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff0880789a0 msgr2=0x7ff08807ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff0880789a0 0x7ff08807ae60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0005880 tx=0x7ff0a0007a20 comp rx=0 tx=0).stop 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 10 monclient: shutdown 2024-07-26T23:35:25.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 20 monclient: shutdown canceling and discarding version request 1 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a414c1d0 msgr2=0x7ff0a4138b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 --2- 172.21.15.139:0/2736713592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a414c1d0 0x7ff0a4138b00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff09c033000 tx=0x7ff09c004a40 comp rx=0 tx=0).stop 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0aa76c640 1 -- 172.21.15.139:0/2736713592 reap_dead start 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 shutdown_connections 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 >> 172.21.15.139:0/2736713592 conn(0x7ff0a40b62c0 msgr2=0x7ff0a40b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 shutdown_connections 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.497+0000 7ff0ac1cf640 1 -- 172.21.15.139:0/2736713592 wait complete. 2024-07-26T23:35:25.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.498+0000 7ff0ac1cf640 1 librados: shutdown 2024-07-26T23:35:25.513 INFO:teuthology.orchestra.run.smithi139.stdout:85899345931 2024-07-26T23:35:25.513 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 15 client.4324.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 20 client.4324.objecter _get_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 20 client.4324.objecter put_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 20 client.4324.objecter _get_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 15 client.4324.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 20 client.4324.objecter get_session s=0x7fa71807e760 osd=5 3 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 15 client.4324.objecter _session_command_op_assign 5 2 2024-07-26T23:35:25.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 20 client.4324.objecter put_session s=0x7fa71807e760 osd=5 4 2024-07-26T23:35:25.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 10 client.4324.objecter _send_command 2 2024-07-26T23:35:25.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.505+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 --> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa7340ad780 con 0x7fa71807ee30 2024-07-26T23:35:25.515 INFO:teuthology.orchestra.run.smithi139.stdout:64424509448 2024-07-26T23:35:25.516 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-07-26T23:35:25.516 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.510+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fa3980405b0 con 0x7fa3b00ace50 2024-07-26T23:35:25.516 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa3a7fff640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa390078b90 0x7fa39007b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.517 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa729ffb640 1 -- 172.21.15.139:0/1917811410 <== osd.5 v2:172.21.15.177:6802/228413488 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa7340ad780 con 0x7fa71807ee30 2024-07-26T23:35:25.517 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa729ffb640 10 client.4324.objecter ms_dispatch 0x7fa734000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.517 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa729ffb640 10 client.4324.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.517 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa729ffb640 20 client.4324.objecter put_session s=0x7fa71807e760 osd=5 4 2024-07-26T23:35:25.517 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa729ffb640 15 client.4324.objecter _session_command_op_remove 5 2 2024-07-26T23:35:25.517 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa3af7fe640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa390078b90 0x7fa39007b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.517 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa3af7fe640 10 monclient: get_auth_request con 0x7fa390078b90 auth_method 0 2024-07-26T23:35:25.518 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 10 librados: watch_flush enter 2024-07-26T23:35:25.518 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 10 librados: watch_flush exit 2024-07-26T23:35:25.518 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 10 client.4324.objecter close_session for osd.5 2024-07-26T23:35:25.518 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7fa71807ee30 msgr2=0x7fa718081270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.518 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7fa71807ee30 0x7fa718081270 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.519 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 20 client.4324.objecter put_session s=0x7fa71807e760 osd=5 1 2024-07-26T23:35:25.519 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 20 client.4324.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.519 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 10 client.4324.objecter successfully canceled tick 2024-07-26T23:35:25.519 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa3a7fff640 10 client.4326.objecter ms_handle_connect 0x7fa390078b90 2024-07-26T23:35:25.519 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa3af7fe640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa390078b90 0x7fa39007b050 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0003b80 tx=0x7fa3a00098d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa7180789f0 msgr2=0x7fa71807aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa7180789f0 0x7fa71807aeb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa73003a000 tx=0x7fa730035000 comp rx=0 tx=0).stop 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 10 monclient: shutdown 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 20 monclient: shutdown canceling and discarding version request 1 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa73413dd40 msgr2=0x7fa73413ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 --2- 172.21.15.139:0/1917811410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa73413dd40 0x7fa73413ebe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa72c001a50 tx=0x7fa72c001f10 comp rx=0 tx=0).stop 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa739fa0640 1 -- 172.21.15.139:0/1917811410 reap_dead start 2024-07-26T23:35:25.520 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 shutdown_connections 2024-07-26T23:35:25.521 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.511+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 >> 172.21.15.139:0/1917811410 conn(0x7fa7340b1c10 msgr2=0x7fa7340b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.521 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.512+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 shutdown_connections 2024-07-26T23:35:25.521 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.512+0000 7fa73a7a1640 1 -- 172.21.15.139:0/1917811410 wait complete. 2024-07-26T23:35:25.521 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.512+0000 7fa73a7a1640 1 librados: shutdown 2024-07-26T23:35:25.521 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.521 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 15 client.4321.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 20 client.4321.objecter _get_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 20 client.4321.objecter put_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 20 client.4321.objecter _get_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 15 client.4321.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 20 client.4321.objecter get_session s=0x7f991807e790 osd=2 3 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 15 client.4321.objecter _session_command_op_assign 2 2 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 20 client.4321.objecter put_session s=0x7f991807e790 osd=2 4 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 10 client.4321.objecter _send_command 2 2024-07-26T23:35:25.522 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.513+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 --> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f99340b3320 con 0x7f991807ee60 2024-07-26T23:35:25.525 INFO:teuthology.orchestra.run.smithi139.stdout:55834574856 2024-07-26T23:35:25.526 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f9929ffb640 1 -- 172.21.15.139:0/975475855 <== osd.2 v2:172.21.15.139:6824/1043169956 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f99340b3320 con 0x7f991807ee60 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f9929ffb640 10 client.4321.objecter ms_dispatch 0x7f9934000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f9929ffb640 10 client.4321.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f9929ffb640 20 client.4321.objecter put_session s=0x7f991807e790 osd=2 4 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f9929ffb640 15 client.4321.objecter _session_command_op_remove 2 2 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 10 librados: watch_flush enter 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 10 librados: watch_flush exit 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 10 client.4321.objecter close_session for osd.2 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f991807ee60 msgr2=0x7f99180812a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.527 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f991807ee60 0x7f99180812a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 20 client.4321.objecter put_session s=0x7f991807e790 osd=2 1 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 20 client.4321.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.523+0000 7f993b71c640 10 client.4321.objecter successfully canceled tick 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9918078a20 msgr2=0x7f991807aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f9918078a20 0x7f991807aee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9930006f40 tx=0x7f9930006e90 comp rx=0 tx=0).stop 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 10 monclient: shutdown 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 20 monclient: shutdown canceling and discarding version request 1 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99340b28d0 msgr2=0x7f9934145160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 --2- 172.21.15.139:0/975475855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99340b28d0 0x7f9934145160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c730 tx=0x7f992c00cbf0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f9939cb9640 1 -- 172.21.15.139:0/975475855 reap_dead start 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 shutdown_connections 2024-07-26T23:35:25.528 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 >> 172.21.15.139:0/975475855 conn(0x7f99340b5b40 msgr2=0x7f99340b4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 shutdown_connections 2024-07-26T23:35:25.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 -- 172.21.15.139:0/975475855 wait complete. 2024-07-26T23:35:25.529 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.524+0000 7f993b71c640 1 librados: shutdown 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.526+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557400ad40 con 0x7f557c0aed00 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.526+0000 7f5571ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.526+0000 7f55737fe640 4 set_mon_vals no callback set 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.527+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557401c640 con 0x7f557c0aed00 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.527+0000 7f5571ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.527+0000 7f5571ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.527+0000 7f5571ffb640 10 monclient: dump: 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.530 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.531 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.531 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.531 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.531 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.531 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.531 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.531 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.532+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f557401a070 con 0x7f557c0aed00 2024-07-26T23:35:25.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.533+0000 7f5571ffb640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f55600789a0 0x7f556007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.533+0000 7f5581000640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f55600789a0 0x7f556007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.534 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.533+0000 7f5581000640 10 monclient: get_auth_request con 0x7f55600789a0 auth_method 0 2024-07-26T23:35:25.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.534+0000 7f5581000640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f55600789a0 0x7f556007ae60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5578000f60 tx=0x7f55780079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.535 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.534+0000 7f5571ffb640 10 client.4336.objecter ms_handle_connect 0x7f55600789a0 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f55740970a0 con 0x7f557c0aed00 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 10 client.4336.objecter ms_dispatch 0x7f557c000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 3 client.4336.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 3 client.4336.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 10 client.4336.objecter checking command 1 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f556007ede0 0x7f5560081220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 20 client.4336.objecter _get_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 20 client.4336.objecter put_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 15 client.4336.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 20 client.4336.objecter _get_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 20 client.4336.objecter get_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 15 client.4336.objecter _session_command_op_assign 4 1 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 20 client.4336.objecter put_session s=0x7f556007e710 osd=4 4 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 10 client.4336.objecter _send_command 1 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 --> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f557400ad40 con 0x7f556007ede0 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 20 client.4336.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f557405dd80 con 0x7f557c0aed00 2024-07-26T23:35:25.538 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.536+0000 7f5571ffb640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-07-26T23:35:25.539 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.537+0000 7f5582002640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f556007ede0 0x7f5560081220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.539 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.537+0000 7f5582002640 10 monclient: get_auth_request con 0x7f556007ede0 auth_method 0 2024-07-26T23:35:25.542 INFO:teuthology.orchestra.run.smithi139.stdout:90194313226 2024-07-26T23:35:25.543 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-07-26T23:35:25.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.542+0000 7f5571ffb640 10 client.4336.objecter ms_handle_connect 0x7f556007ede0 2024-07-26T23:35:25.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.542+0000 7f5582002640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f556007ede0 0x7f5560081220 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.563 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.561+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fc4180453f0 con 0x7fc424143d00 2024-07-26T23:35:25.563 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.562+0000 7fc412ffd640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fc4040789a0 0x7fc40407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.563 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.562+0000 7fc42259c640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fc4040789a0 0x7fc40407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.563 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.563+0000 7fc42259c640 10 monclient: get_auth_request con 0x7fc4040789a0 auth_method 0 2024-07-26T23:35:25.563 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.563+0000 7fc42259c640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fc4040789a0 0x7fc40407ae60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc414009890 tx=0x7fc414008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.564 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.563+0000 7fc412ffd640 10 client.4332.objecter ms_handle_connect 0x7fc4040789a0 2024-07-26T23:35:25.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f88700b8e30 con 0x7f887813c6f0 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 10 client.4335.objecter ms_dispatch 0x7f8878000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 3 client.4335.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 3 client.4335.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 10 client.4335.objecter checking command 1 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f886407ede0 0x7f8864081220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 20 client.4335.objecter _get_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 20 client.4335.objecter put_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 15 client.4335.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 20 client.4335.objecter _get_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 20 client.4335.objecter get_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 15 client.4335.objecter _session_command_op_assign 1 1 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 20 client.4335.objecter put_session s=0x7f886407e710 osd=1 4 2024-07-26T23:35:25.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 10 client.4335.objecter _send_command 1 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f887003be90 con 0x7f886407ede0 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 20 client.4335.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f887007fae0 con 0x7f887813c6f0 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7f885ffff640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fa39807fe00 con 0x7fa3b00ace50 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7fa3a7fff640 10 client.4326.objecter ms_dispatch 0x7fa3b0000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7fa3a7fff640 3 client.4326.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7fa3a7fff640 3 client.4326.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.583+0000 7fa3a7fff640 10 client.4326.objecter checking command 1 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7fa39007efd0 0x7fa390081410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 20 client.4326.objecter _get_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 20 client.4326.objecter put_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 15 client.4326.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 20 client.4326.objecter _get_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 20 client.4326.objecter get_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 15 client.4326.objecter _session_command_op_assign 0 1 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 20 client.4326.objecter put_session s=0x7fa39007e900 osd=0 4 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 10 client.4326.objecter _send_command 1 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 --> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa39802ebe0 con 0x7fa39007efd0 2024-07-26T23:35:25.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 20 client.4326.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa3980801c0 con 0x7fa3b00ace50 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.584+0000 7fa3a7fff640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fc4180b9150 con 0x7fc424143d00 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 10 client.4332.objecter ms_dispatch 0x7fc424000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 3 client.4332.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 3 client.4332.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 10 client.4332.objecter checking command 1 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7fc40407ede0 0x7fc404081220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 20 client.4332.objecter _get_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 20 client.4332.objecter put_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 15 client.4332.objecter _session_command_op_remove -1 1 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 20 client.4332.objecter _get_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 20 client.4332.objecter get_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 15 client.4332.objecter _session_command_op_assign 6 1 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 20 client.4332.objecter put_session s=0x7fc40407e710 osd=6 4 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 10 client.4332.objecter _send_command 1 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 --> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc41803ae90 con 0x7fc40407ede0 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 20 client.4332.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc41807ffd0 con 0x7fc424143d00 2024-07-26T23:35:25.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fc412ffd640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-07-26T23:35:25.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7f8877fff640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f886407ede0 0x7f8864081220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7f8877fff640 10 monclient: get_auth_request con 0x7f886407ede0 auth_method 0 2024-07-26T23:35:25.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fa3affff640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7fa39007efd0 0x7fa390081410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.585+0000 7fa3affff640 10 monclient: get_auth_request con 0x7fa39007efd0 auth_method 0 2024-07-26T23:35:25.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.586+0000 7fc422d9d640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7fc40407ede0 0x7fc404081220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.586+0000 7fc422d9d640 10 monclient: get_auth_request con 0x7fc40407ede0 auth_method 0 2024-07-26T23:35:25.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.589+0000 7fc412ffd640 10 client.4332.objecter ms_handle_connect 0x7fc40407ede0 2024-07-26T23:35:25.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.589+0000 7fc422d9d640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7fc40407ede0 0x7fc404081220 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.591 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.591+0000 7fa3a7fff640 10 client.4326.objecter ms_handle_connect 0x7fa39007efd0 2024-07-26T23:35:25.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.591+0000 7fa3affff640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7fa39007efd0 0x7fa390081410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.593+0000 7f8877fff640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f886407ede0 0x7f8864081220 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.593+0000 7f885ffff640 10 client.4335.objecter ms_handle_connect 0x7f886407ede0 2024-07-26T23:35:25.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.618+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== osd.4 v2:172.21.15.177:6826/4112782318 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f557400ad40 con 0x7f556007ede0 2024-07-26T23:35:25.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.618+0000 7f5571ffb640 10 client.4336.objecter ms_dispatch 0x7f557c000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.618+0000 7f5571ffb640 10 client.4336.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.618+0000 7f5571ffb640 20 client.4336.objecter put_session s=0x7f556007e710 osd=4 4 2024-07-26T23:35:25.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.618+0000 7f5571ffb640 15 client.4336.objecter _session_command_op_remove 4 1 2024-07-26T23:35:25.642 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.641+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== osd.6 v2:172.21.15.177:6810/1761005298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc41803ae90 con 0x7fc40407ede0 2024-07-26T23:35:25.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.641+0000 7fc412ffd640 10 client.4332.objecter ms_dispatch 0x7fc424000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.641+0000 7fc412ffd640 10 client.4332.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.641+0000 7fc412ffd640 20 client.4332.objecter put_session s=0x7fc40407e710 osd=6 4 2024-07-26T23:35:25.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.641+0000 7fc412ffd640 15 client.4332.objecter _session_command_op_remove 6 1 2024-07-26T23:35:25.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 10 client.4336.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 15 client.4336.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 20 client.4336.objecter _get_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 20 client.4336.objecter put_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 20 client.4336.objecter _get_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 15 client.4336.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 20 client.4336.objecter get_session s=0x7f556007e710 osd=4 3 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 15 client.4336.objecter _session_command_op_assign 4 2 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 20 client.4336.objecter put_session s=0x7f556007e710 osd=4 4 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 10 client.4336.objecter _send_command 2 2024-07-26T23:35:25.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.648+0000 7f5582803640 1 -- 172.21.15.139:0/4059357468 --> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f557c13f6a0 con 0x7f556007ede0 2024-07-26T23:35:25.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.657+0000 7f160b154640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.657+0000 7f160b154640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.676 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.657+0000 7f160b154640 10 monclient: monmap: 2024-07-26T23:35:25.676 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.676 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.676 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.657692+0000 2024-07-26T23:35:25.676 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.657692+0000 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.658+0000 7f160b154640 1 Processor -- start 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.658+0000 7f160b154640 1 -- start start 2024-07-26T23:35:25.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.658+0000 7f160b154640 10 monclient: init 2024-07-26T23:35:25.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f5571ffb640 1 -- 172.21.15.139:0/4059357468 <== osd.4 v2:172.21.15.177:6826/4112782318 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f557c13f6a0 con 0x7f556007ede0 2024-07-26T23:35:25.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f5571ffb640 10 client.4336.objecter ms_dispatch 0x7f557c000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f5571ffb640 10 client.4336.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f5571ffb640 20 client.4336.objecter put_session s=0x7f556007e710 osd=4 4 2024-07-26T23:35:25.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f5571ffb640 15 client.4336.objecter _session_command_op_remove 4 2 2024-07-26T23:35:25.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f55677fe640 10 librados: watch_flush enter 2024-07-26T23:35:25.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f55677fe640 10 librados: watch_flush exit 2024-07-26T23:35:25.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f55677fe640 10 client.4336.objecter close_session for osd.4 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.659+0000 7f55677fe640 1 -- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f556007ede0 msgr2=0x7f5560081220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f556007ede0 0x7f5560081220 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 20 client.4336.objecter put_session s=0x7f556007e710 osd=4 1 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 20 client.4336.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 10 client.4336.objecter successfully canceled tick 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 1 -- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f55600789a0 msgr2=0x7f556007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f55600789a0 0x7f556007ae60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5578000f60 tx=0x7f55780079b0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 10 monclient: shutdown 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 1 -- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f557c0aed00 msgr2=0x7f557c139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 1 --2- 172.21.15.139:0/4059357468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f557c0aed00 0x7f557c139b60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f557400cbc0 tx=0x7f557400a430 comp rx=0 tx=0).stop 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f5582002640 1 -- 172.21.15.139:0/4059357468 reap_dead start 2024-07-26T23:35:25.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 1 -- 172.21.15.139:0/4059357468 shutdown_connections 2024-07-26T23:35:25.682 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.660+0000 7f55677fe640 1 -- 172.21.15.139:0/4059357468 >> 172.21.15.139:0/4059357468 conn(0x7f557c0b1c10 msgr2=0x7f557c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.682 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.661+0000 7f55677fe640 1 -- 172.21.15.139:0/4059357468 shutdown_connections 2024-07-26T23:35:25.682 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.661+0000 7f55677fe640 1 -- 172.21.15.139:0/4059357468 wait complete. 2024-07-26T23:35:25.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.661+0000 7f160b154640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.661+0000 7f160b154640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-07-26T23:35:25.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.661+0000 7f160b154640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16041470c0 0x7f1604140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.661+0000 7f160b154640 10 monclient(hunting): picked mon.noname-c con 0x7f16041470c0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f55677fe640 1 librados: shutdown 2024-07-26T23:35:25.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1604140e90 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f16040b48d0 0x7f160414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 10 monclient(hunting): picked mon.noname-a con 0x7f16040b48d0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1604147ce0 con 0x7f1604140e90 2024-07-26T23:35:25.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f1608ef0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16041470c0 0x7f1604140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f16040acaf0 con 0x7f16040b48d0 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f16041477c0 con 0x7f16041470c0 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.663+0000 7f1608ef0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16041470c0 0x7f1604140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41314/0 (socket says 172.21.15.139:41314) 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.663+0000 7f1608ef0640 1 -- 172.21.15.139:0/3053056925 learned_addr learned my addr 172.21.15.139:0/3053056925 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.662+0000 7f160b154640 10 monclient(hunting): authenticate will time out at 1358.697979s 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.663+0000 7f1608ef0640 10 monclient(hunting): get_auth_request con 0x7f16041470c0 auth_method 0 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.663+0000 7f1608ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.663+0000 7f1608ef0640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.663+0000 7f1608ef0640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.665+0000 7f16096f1640 1 --2- 172.21.15.139:0/3053056925 >> v2:172.21.15.139:3300/0 conn(0x7f16040b48d0 0x7f160414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.665+0000 7f16096f1640 10 monclient(hunting): get_auth_request con 0x7f16040b48d0 auth_method 0 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.665+0000 7f16096f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.665+0000 7f16096f1640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.665+0000 7f16096f1640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.668+0000 7f16096f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.668+0000 7f16096f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.668+0000 7f16096f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.666+0000 7f3ea08b3640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.666+0000 7f3ea08b3640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.666+0000 7f3ea08b3640 10 monclient: monmap: 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.666900+0000 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.666900+0000 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.688 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.668+0000 7f1608ef0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.668+0000 7f1608ef0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.668+0000 7f1608ef0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 1 -- 172.21.15.139:0/3053056925 >> v2:172.21.15.139:3300/0 conn(0x7f16040b48d0 msgr2=0x7f160414e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 1 --2- 172.21.15.139:0/3053056925 >> v2:172.21.15.139:3300/0 conn(0x7f16040b48d0 0x7f160414e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 1 -- 172.21.15.139:0/3053056925 >> v1:172.21.15.139:6789/0 conn(0x7f1604140e90 legacy=0x7f16040b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 1 monclient: found mon.noname-c 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.670+0000 7f1608ef0640 1 -- 172.21.15.139:0/3053056925 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16041474a0 con 0x7f16041470c0 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7f3ea08b3640 1 Processor -- start 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 10 client.4332.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 15 client.4332.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 20 client.4332.objecter _get_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 20 client.4332.objecter put_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 20 client.4332.objecter _get_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 15 client.4332.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 20 client.4332.objecter get_session s=0x7fc40407e710 osd=6 3 2024-07-26T23:35:25.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 15 client.4332.objecter _session_command_op_assign 6 2 2024-07-26T23:35:25.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7fc428f9d640 20 client.4332.objecter put_session s=0x7fc40407e710 osd=6 4 2024-07-26T23:35:25.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7fc428f9d640 10 client.4332.objecter _send_command 2 2024-07-26T23:35:25.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7fc428f9d640 1 -- 172.21.15.139:0/4273215465 --> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc4240771c0 con 0x7fc40407ede0 2024-07-26T23:35:25.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7f3ea08b3640 1 -- start start 2024-07-26T23:35:25.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.672+0000 7f3ea08b3640 10 monclient: init 2024-07-26T23:35:25.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c0b1300 0x7f3e9c0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9c0b1300 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e9c13ca20 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f3e9c13ce20 0x7f3e9c143430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9c13ce20 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e9c0af2e0 con 0x7f3e9c13ca20 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f3e9c0b16e0 con 0x7f3e9c13ce20 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9c0ae5c0 con 0x7f3e9c0b1300 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f3ea08b3640 10 monclient(hunting): authenticate will time out at 1358.708643s 2024-07-26T23:35:25.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.673+0000 7f1608ef0640 1 --2- 172.21.15.139:0/3053056925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041470c0 0x7f1604140950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f160406efe0 tx=0x7f15f802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c82a9d89ab18fd0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.674+0000 7f3e9b7fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c0b1300 0x7f3e9c0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9b7fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c0b1300 0x7f3e9c0aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60718/0 (socket says 172.21.15.139:60718) 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9b7fe640 1 -- 172.21.15.139:0/2126911040 learned_addr learned my addr 172.21.15.139:0/2126911040 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9b7fe640 10 monclient(hunting): get_auth_request con 0x7f3e9c0b1300 auth_method 0 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9bfff640 1 --2- 172.21.15.139:0/2126911040 >> v2:172.21.15.139:3300/0 conn(0x7f3e9c13ce20 0x7f3e9c143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9bfff640 10 monclient(hunting): get_auth_request con 0x7f3e9c13ce20 auth_method 0 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.675+0000 7f3e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.676+0000 7f3e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.676+0000 7f3e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.696 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.676+0000 7f3e9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 1 -- 172.21.15.139:0/3053056925 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8034070 con 0x7f16041470c0 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: dump: 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 1 -- 172.21.15.139:0/3053056925 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f8030610 con 0x7f16041470c0 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f160b154640 5 monclient: authenticate success, global_id 4339 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f160b154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:25.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 1 -- 172.21.15.139:0/3053056925 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8030a20 con 0x7f16041470c0 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f160b154640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f160b154640 4 set_mon_vals no callback set 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f16037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f16037fe640 10 monclient: dump: 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.699 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.700 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.700 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.700 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.700 INFO:teuthology.orchestra.run.smithi139.stdout:77309411338 2024-07-26T23:35:25.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f3e9b7fe640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f3e9b7fe640 1 -- 172.21.15.139:0/2126911040 >> v2:172.21.15.139:3300/0 conn(0x7f3e9c13ce20 msgr2=0x7f3e9c143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f3e9b7fe640 1 --2- 172.21.15.139:0/2126911040 >> v2:172.21.15.139:3300/0 conn(0x7f3e9c13ce20 0x7f3e9c143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.678+0000 7f3e9b7fe640 1 -- 172.21.15.139:0/2126911040 >> v1:172.21.15.139:6789/0 conn(0x7f3e9c13ca20 legacy=0x7f3e9c142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f3e9b7fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f3e9b7fe640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f3e9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f3e9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f3e9b7fe640 1 -- 172.21.15.139:0/2126911040 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c0ae8a0 con 0x7f3e9c0b1300 2024-07-26T23:35:25.702 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-07-26T23:35:25.702 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f160b154640 10 monclient: shutdown 2024-07-26T23:35:25.702 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f160b154640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7fc412ffd640 1 -- 172.21.15.139:0/4273215465 <== osd.6 v2:172.21.15.177:6810/1761005298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc4240771c0 con 0x7fc40407ede0 2024-07-26T23:35:25.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7fc412ffd640 10 client.4332.objecter ms_dispatch 0x7fc424000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7fc412ffd640 10 client.4332.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7fc412ffd640 20 client.4332.objecter put_session s=0x7fc40407e710 osd=6 4 2024-07-26T23:35:25.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7fc412ffd640 15 client.4332.objecter _session_command_op_remove 6 2 2024-07-26T23:35:25.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.679+0000 7f160b154640 1 -- 172.21.15.139:0/3053056925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041470c0 msgr2=0x7f1604140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 10 librados: watch_flush enter 2024-07-26T23:35:25.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 10 librados: watch_flush exit 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 10 client.4332.objecter close_session for osd.6 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 -- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7fc40407ede0 msgr2=0x7fc404081220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7fc40407ede0 0x7fc404081220 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 20 client.4332.objecter put_session s=0x7fc40407e710 osd=6 1 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 20 client.4332.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 10 client.4332.objecter successfully canceled tick 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 -- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fc4040789a0 msgr2=0x7fc40407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fc4040789a0 0x7fc40407ae60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc414009890 tx=0x7fc414008040 comp rx=0 tx=0).stop 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 10 monclient: shutdown 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 -- 172.21.15.139:0/4273215465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc424143d00 msgr2=0x7fc4240b4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 --2- 172.21.15.139:0/4273215465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc424143d00 0x7fc4240b4360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc418009720 tx=0x7fc4180046c0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.706 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc422d9d640 1 -- 172.21.15.139:0/4273215465 reap_dead start 2024-07-26T23:35:25.707 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 -- 172.21.15.139:0/4273215465 shutdown_connections 2024-07-26T23:35:25.707 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.680+0000 7fc410ff9640 1 -- 172.21.15.139:0/4273215465 >> 172.21.15.139:0/4273215465 conn(0x7fc4240b9c10 msgr2=0x7fc424146320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.707 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.681+0000 7f160b154640 1 --2- 172.21.15.139:0/3053056925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041470c0 0x7f1604140950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f160406efe0 tx=0x7f15f802b960 comp rx=0 tx=0).stop 2024-07-26T23:35:25.707 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.681+0000 7fc410ff9640 1 -- 172.21.15.139:0/4273215465 shutdown_connections 2024-07-26T23:35:25.707 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.681+0000 7fc410ff9640 1 -- 172.21.15.139:0/4273215465 wait complete. 2024-07-26T23:35:25.708 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.681+0000 7f160b154640 1 -- 172.21.15.139:0/3053056925 shutdown_connections 2024-07-26T23:35:25.708 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.681+0000 7fc410ff9640 1 librados: shutdown 2024-07-26T23:35:25.708 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.681+0000 7f160b154640 1 --2- 172.21.15.139:0/3053056925 >> v2:172.21.15.139:3300/0 conn(0x7f16040b48d0 0x7f160414e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.708 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.682+0000 7f160b154640 1 --2- 172.21.15.139:0/3053056925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16041470c0 0x7f1604140950 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f160406efe0 tx=0x7f15f802b960 comp rx=0 tx=0).stop 2024-07-26T23:35:25.709 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.682+0000 7f160b154640 1 -- 172.21.15.139:0/3053056925 >> 172.21.15.139:0/3053056925 conn(0x7f16040b62c0 msgr2=0x7f16040b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.709 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.682+0000 7fa521c7b640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.709 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.682+0000 7f160b154640 1 -- 172.21.15.139:0/3053056925 shutdown_connections 2024-07-26T23:35:25.709 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.682+0000 7f160b154640 1 -- 172.21.15.139:0/3053056925 wait complete. 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.683+0000 7f160b154640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.683+0000 7f160b154640 10 monclient: monmap: 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.710 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.683+0000 7f3e9b7fe640 1 --2- 172.21.15.139:0/2126911040 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c0b1300 0x7f3e9c0aeda0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4c74d719e4175d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.682+0000 7fa521c7b640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.682+0000 7fa521c7b640 10 monclient: monmap: 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.682448+0000 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.682448+0000 2024-07-26T23:35:25.711 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.712 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.712 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.712 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.712 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.712 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.712 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.712 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7f160b154640 1 librados: starting msgr at 2024-07-26T23:35:25.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7f160b154640 1 librados: starting objecter 2024-07-26T23:35:25.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7f160b154640 1 Processor -- start 2024-07-26T23:35:25.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7f160b154640 1 -- start start 2024-07-26T23:35:25.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7f160b154640 1 librados: setting wanted keys 2024-07-26T23:35:25.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7f160b154640 1 librados: calling monclient init 2024-07-26T23:35:25.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7f160b154640 10 monclient: init 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7fa521c7b640 1 Processor -- start 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7fa521c7b640 1 -- start start 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.684+0000 7fa521c7b640 10 monclient: init 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa514030880 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fa514031a20 0x7fa5140331a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa514031a20 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5140c45e0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5140324e0 con 0x7fa514030880 2024-07-26T23:35:25.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5140327c0 con 0x7fa5140c45e0 2024-07-26T23:35:25.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fa514030c60 con 0x7fa514031a20 2024-07-26T23:35:25.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.685+0000 7fa521c7b640 10 monclient(hunting): authenticate will time out at 1358.720769s 2024-07-26T23:35:25.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7f160b154640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7f160b154640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:25.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7f160b154640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16040b48d0 0x7f160413e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7f160b154640 10 monclient(hunting): picked mon.noname-b con 0x7f16040b48d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7f160b154640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1604140e90 0x7f1604137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7f160b154640 10 monclient(hunting): picked mon.noname-a con 0x7f1604140e90 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f160413d5b0 0x7f1604137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 10 monclient(hunting): picked mon.noname-c con 0x7f160413d5b0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f1604141410 con 0x7f1604140e90 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1604074850 con 0x7f16040b48d0 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7fa52147a640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fa5140c45e0 0x7fa5140c4a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:52018/0 (socket says 172.21.15.139:52018) 2024-07-26T23:35:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.686+0000 7fa52147a640 1 -- 172.21.15.139:0/1499583124 learned_addr learned my addr 172.21.15.139:0/1499583124 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:25.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f160414e720 con 0x7f160413d5b0 2024-07-26T23:35:25.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.687+0000 7f160b154640 10 monclient(hunting): authenticate will time out at 1358.722854s 2024-07-26T23:35:25.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.688+0000 7fa51bfff640 1 --2- 172.21.15.139:0/1499583124 >> v2:172.21.15.139:3300/0 conn(0x7fa514031a20 0x7fa5140331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.688+0000 7fa51b7fe640 1 -- 172.21.15.139:0/1499583124 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343985441 0 0) 0x7fa5140327c0 con 0x7fa5140c45e0 2024-07-26T23:35:25.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.688+0000 7fa51b7fe640 10 monclient(hunting): _negotiate 2024-07-26T23:35:25.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.688+0000 7fa51b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.688+0000 7fa51b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f1608ef0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16040b48d0 0x7f160413e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.688+0000 7fa51b7fe640 10 monclient(hunting): my global_id is 4342 2024-07-26T23:35:25.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f1603fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1604140e90 0x7f1604137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.688+0000 7fa51b7fe640 1 -- 172.21.15.139:0/1499583124 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa50c005730 con 0x7fa5140c45e0 2024-07-26T23:35:25.721 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f16096f1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f160413d5b0 0x7f1604137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.721 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f16096f1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f160413d5b0 0x7f1604137d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60734/0 (socket says 172.21.15.139:60734) 2024-07-26T23:35:25.721 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f16096f1640 1 -- 172.21.15.139:0/1013571710 learned_addr learned my addr 172.21.15.139:0/1013571710 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.721 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f16096f1640 10 monclient(hunting): get_auth_request con 0x7f160413d5b0 auth_method 0 2024-07-26T23:35:25.721 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f16096f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.721 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f16096f1640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f16096f1640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f1608ef0640 10 monclient(hunting): get_auth_request con 0x7f16040b48d0 auth_method 0 2024-07-26T23:35:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f1608ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f1608ef0640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.689+0000 7f1608ef0640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7fa51bfff640 10 monclient(hunting): get_auth_request con 0x7fa514031a20 auth_method 0 2024-07-26T23:35:25.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7fa51bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7fa51bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7fa51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f1608ef0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f1608ef0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f1608ef0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 1 -- 172.21.15.139:0/2126911040 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0317c0 con 0x7f3e9c0b1300 2024-07-26T23:35:25.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: dump: 2024-07-26T23:35:25.724 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 1 -- 172.21.15.139:0/2126911040 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8c037070 con 0x7f3e9c0b1300 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 1 -- 172.21.15.139:0/2126911040 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c031b80 con 0x7f3e9c0b1300 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.690+0000 7f3e9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.691+0000 7f3e9a7fc640 10 monclient: dump: 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.726 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.727 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.727 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.691+0000 7f16096f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.691+0000 7f16096f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.691+0000 7f16096f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.691+0000 7f3ea08b3640 5 monclient: authenticate success, global_id 4289 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.691+0000 7f3ea08b3640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.691+0000 7f3ea08b3640 4 set_mon_vals no callback set 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 10 monclient: shutdown 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2126911040 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c0b1300 msgr2=0x7f3e9c0aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2126911040 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c0b1300 0x7f3e9c0aeda0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c02b860 comp rx=0 tx=0).stop 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2126911040 shutdown_connections 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2126911040 >> v2:172.21.15.139:3300/0 conn(0x7f3e9c13ce20 0x7f3e9c143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2126911040 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c0b1300 0x7f3e9c0aeda0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2126911040 >> 172.21.15.139:0/2126911040 conn(0x7f3e9c0b1c10 msgr2=0x7f3e9c0b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.729 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7fa51bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.729 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7fa51bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.729 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.692+0000 7fa51bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.729 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2126911040 shutdown_connections 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 1 -- 172.21.15.139:0/1499583124 >> v1:172.21.15.177:6789/0 conn(0x7fa5140c45e0 legacy=0x7fa5140c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 1 -- 172.21.15.139:0/1499583124 >> v1:172.21.15.139:6789/0 conn(0x7fa514030880 legacy=0x7fa514031310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7fa51bfff640 1 -- 172.21.15.139:0/1499583124 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa514032aa0 con 0x7fa514031a20 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.694+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2126911040 wait complete. 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.695+0000 7f3ea08b3640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.695+0000 7f3ea08b3640 10 monclient: monmap: 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.731 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.695+0000 7f3ea08b3640 1 librados: starting msgr at 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.695+0000 7f3ea08b3640 1 librados: starting objecter 2024-07-26T23:35:25.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.695+0000 7f3ea08b3640 1 Processor -- start 2024-07-26T23:35:25.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.695+0000 7f1608ef0640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-07-26T23:35:25.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 1 -- 172.21.15.139:0/1013571710 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f160413d5b0 msgr2=0x7f1604137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f160413d5b0 0x7f1604137d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 1 -- 172.21.15.139:0/1013571710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1604140e90 msgr2=0x7f1604137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1604140e90 0x7f1604137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f1608ef0640 1 -- 172.21.15.139:0/1013571710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160413fcf0 con 0x7f16040b48d0 2024-07-26T23:35:25.735 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f3ea08b3640 1 -- start start 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f3ea08b3640 1 librados: setting wanted keys 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f3ea08b3640 1 librados: calling monclient init 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.696+0000 7f3ea08b3640 10 monclient: init 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e9c0b1300 0x7f3e9c1403c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9c0b1300 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c13ca20 0x7f3e9c146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9c13ca20 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e9c13ce20 0x7f3e9c139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9c13ce20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c0b28b0 con 0x7f3e9c0b1300 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c0b2730 con 0x7f3e9c13ce20 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9c0b2a30 con 0x7f3e9c13ca20 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.697+0000 7f3ea08b3640 10 monclient(hunting): authenticate will time out at 1358.732790s 2024-07-26T23:35:25.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.698+0000 7fa51bfff640 1 --2- 172.21.15.139:0/1499583124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514031a20 0x7fa5140331a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa5080097a0 tx=0x7fa50802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fba6ab0f07dc923 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.698+0000 7f1608ef0640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16040b48d0 0x7f160413e1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1604147940 tx=0x7f15f8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.698+0000 7f16017fa640 10 client.?.objecter ms_handle_connect 0x7f16040b48d0 2024-07-26T23:35:25.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.698+0000 7f16017fa640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.739 INFO:teuthology.orchestra.run.smithi139.stdout:55834574856 2024-07-26T23:35:25.739 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.701+0000 7f3e9bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e9c13ce20 0x7f3e9c139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.701+0000 7f3e9affd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c13ca20 0x7f3e9c146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.702+0000 7f3e9affd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c13ca20 0x7f3e9c146780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60744/0 (socket says 172.21.15.139:60744) 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.702+0000 7f3e9affd640 1 -- 172.21.15.139:0/2846045856 learned_addr learned my addr 172.21.15.139:0/2846045856 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.702+0000 7f3e9affd640 10 monclient(hunting): get_auth_request con 0x7f3e9c13ca20 auth_method 0 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.702+0000 7f3e9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.702+0000 7f3e9affd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.702+0000 7f3e9affd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9bfff640 10 monclient(hunting): get_auth_request con 0x7f3e9c13ce20 auth_method 0 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.703+0000 7f3e9b7fe640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e9c0b1300 0x7f3e9c1403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== osd.1 v2:172.21.15.139:6800/3107328002 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f887003be90 con 0x7f886407ede0 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f885ffff640 10 client.4335.objecter ms_dispatch 0x7f8878000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f885ffff640 10 client.4335.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f885ffff640 20 client.4335.objecter put_session s=0x7f886407e710 osd=1 4 2024-07-26T23:35:25.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f885ffff640 15 client.4335.objecter _session_command_op_remove 1 1 2024-07-26T23:35:25.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f3e9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f3e9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.704+0000 7f3e9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.744 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 1 -- 172.21.15.139:0/1013571710 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8034040 con 0x7f16040b48d0 2024-07-26T23:35:25.744 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.744 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.744 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: dump: 2024-07-26T23:35:25.744 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.744 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 1 -- 172.21.15.139:0/1013571710 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f8030410 con 0x7f16040b48d0 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 1 -- 172.21.15.139:0/1013571710 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8039cd0 con 0x7f16040b48d0 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f16017fa640 10 monclient: dump: 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.746 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f1602ffd640 4 set_mon_vals no callback set 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 5 monclient: authenticate success, global_id 4345 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 10 monclient: _renew_subs 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16041382d0 con 0x7f16040b48d0 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 10 monclient: _renew_subs 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f160413e6e0 con 0x7f16040b48d0 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.705+0000 7f160b154640 1 librados: init done 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.706+0000 7f160b154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.706+0000 7f160b154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.706+0000 7f160b154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.706+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1604140950 con 0x7f16040b48d0 2024-07-26T23:35:25.748 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 1 -- 172.21.15.139:0/1499583124 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080315a0 con 0x7fa514031a20 2024-07-26T23:35:25.748 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.748 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.748 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: dump: 2024-07-26T23:35:25.748 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa521c7b640 5 monclient: authenticate success, global_id 4347 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa521c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 1 -- 172.21.15.139:0/1499583124 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa508037070 con 0x7fa514031a20 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa521c7b640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa521c7b640 4 set_mon_vals no callback set 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== osd.0 v2:172.21.15.139:6808/2123669081 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa39802ebe0 con 0x7fa39007efd0 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa3a7fff640 10 client.4326.objecter ms_dispatch 0x7fa3b0000b90 command_reply(tid 1: 0 ) 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa3a7fff640 10 client.4326.objecter _finish_command 1 = system:0 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa3a7fff640 20 client.4326.objecter put_session s=0x7fa39007e900 osd=0 4 2024-07-26T23:35:25.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa3a7fff640 15 client.4326.objecter _session_command_op_remove 0 1 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 1 -- 172.21.15.139:0/1499583124 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508031960 con 0x7fa514031a20 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7fa51b7fe640 10 monclient: dump: 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.751 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 1 -- 172.21.15.139:0/2846045856 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c13ca20 msgr2=0x7f3e9c146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c13ca20 0x7f3e9c146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.752 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 1 -- 172.21.15.139:0/2846045856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e9c0b1300 msgr2=0x7f3e9c1403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e9c0b1300 0x7f3e9c1403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.753 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.707+0000 7f3e9bfff640 1 -- 172.21.15.139:0/2846045856 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c145dc0 con 0x7f3e9c13ce20 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 10 monclient: shutdown 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 1 -- 172.21.15.139:0/1499583124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514031a20 msgr2=0x7fa5140331a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 1 --2- 172.21.15.139:0/1499583124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514031a20 0x7fa5140331a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa5080097a0 tx=0x7fa50802b840 comp rx=0 tx=0).stop 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 1 -- 172.21.15.139:0/1499583124 shutdown_connections 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 1 --2- 172.21.15.139:0/1499583124 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514031a20 0x7fa5140331a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 1 -- 172.21.15.139:0/1499583124 >> 172.21.15.139:0/1499583124 conn(0x7fa514006010 msgr2=0x7fa514006420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 1 -- 172.21.15.139:0/1499583124 shutdown_connections 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.708+0000 7fa521c7b640 1 -- 172.21.15.139:0/1499583124 wait complete. 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.709+0000 7fa521c7b640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.755 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.709+0000 7fa521c7b640 10 monclient: monmap: 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.709+0000 7fa521c7b640 1 librados: starting msgr at 2024-07-26T23:35:25.756 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.709+0000 7fa521c7b640 1 librados: starting objecter 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.709+0000 7fa521c7b640 1 Processor -- start 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 1 -- start start 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 1 librados: setting wanted keys 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 1 librados: calling monclient init 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 10 monclient: init 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514030880 0x7fa5140c0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.710+0000 7fa521c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa514030880 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa514031a20 0x7fa5140c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa514031a20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5140c45e0 0x7fa5140c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5140c45e0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa514030e80 con 0x7fa514030880 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa51403a680 con 0x7fa514031a20 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa51403a800 con 0x7fa5140c45e0 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa521c7b640 10 monclient(hunting): authenticate will time out at 1358.746065s 2024-07-26T23:35:25.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7f3e9bfff640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e9c13ce20 0x7f3e9c139200 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000d3e0 tx=0x7f3e9000d8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7f3e83fff640 10 client.?.objecter ms_handle_connect 0x7f3e9c13ce20 2024-07-26T23:35:25.759 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7f3e83fff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.711+0000 7fa520c79640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514030880 0x7fa5140c0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa52147a640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5140c45e0 0x7fa5140c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa520c79640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514030880 0x7fa5140c0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:32770/0 (socket says 172.21.15.139:32770) 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa520c79640 1 -- 172.21.15.139:0/2746668543 learned_addr learned my addr 172.21.15.139:0/2746668543 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa520c79640 10 monclient(hunting): get_auth_request con 0x7fa514030880 auth_method 0 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa520c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa520c79640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa520c79640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa52147a640 10 monclient(hunting): get_auth_request con 0x7fa5140c45e0 auth_method 0 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa52147a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa52147a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.712+0000 7fa52147a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.713+0000 7fa520c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.713+0000 7fa520c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.713+0000 7fa520c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.713+0000 7fa52147a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.713+0000 7fa52147a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.713+0000 7fa52147a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 1 -- 172.21.15.139:0/2746668543 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5140c45e0 msgr2=0x7fa5140c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5140c45e0 0x7fa5140c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 1 -- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa514031a20 msgr2=0x7fa5140c0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa514031a20 0x7fa5140c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 1 monclient: found mon.noname-a 2024-07-26T23:35:25.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.715+0000 7fa520c79640 1 -- 172.21.15.139:0/2746668543 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5140c7d30 con 0x7fa514030880 2024-07-26T23:35:25.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.716+0000 7f0547522640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.716+0000 7f0547522640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.716+0000 7f0547522640 10 monclient: monmap: 2024-07-26T23:35:25.765 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.765 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.716294+0000 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.716294+0000 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.717+0000 7f0547522640 1 Processor -- start 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.717+0000 7f0547522640 1 -- start start 2024-07-26T23:35:25.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.717+0000 7f0547522640 10 monclient: init 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0540068740 0x7f0540140940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): picked mon.noname-c con 0x7f0540068740 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540140e80 0x7f0540146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): picked mon.noname-b con 0x7f0540140e80 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f0540147410 0x7f05400b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): picked mon.noname-a con 0x7f0540147410 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f0540068b20 con 0x7f0540147410 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f054006dec0 con 0x7f0540068740 2024-07-26T23:35:25.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f05401412c0 con 0x7f0540140e80 2024-07-26T23:35:25.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.718+0000 7f0547522640 10 monclient(hunting): authenticate will time out at 1358.753826s 2024-07-26T23:35:25.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.719+0000 7fa520c79640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514030880 0x7fa5140c0850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa51000c410 tx=0x7fa51000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.719+0000 7fa5197fa640 10 client.?.objecter ms_handle_connect 0x7fa514030880 2024-07-26T23:35:25.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.720+0000 7fa5197fa640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.722+0000 7f0544abd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540140e80 0x7f0540146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.722+0000 7f0544abd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540140e80 0x7f0540146ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60766/0 (socket says 172.21.15.139:60766) 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.722+0000 7f0544abd640 1 -- 172.21.15.139:0/3914976565 learned_addr learned my addr 172.21.15.139:0/3914976565 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.722+0000 7f0544abd640 10 monclient(hunting): get_auth_request con 0x7f0540140e80 auth_method 0 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.722+0000 7f0544abd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.722+0000 7f0544abd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.722+0000 7f0544abd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.724+0000 7f0544abd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.724+0000 7f0544abd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.724+0000 7f0544abd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 1 -- 172.21.15.139:0/3914976565 >> v2:172.21.15.177:3300/0 conn(0x7f0540068740 msgr2=0x7f0540140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 1 --2- 172.21.15.139:0/3914976565 >> v2:172.21.15.177:3300/0 conn(0x7f0540068740 0x7f0540140940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 1 -- 172.21.15.139:0/3914976565 >> v2:172.21.15.139:3300/0 conn(0x7f0540147410 msgr2=0x7f05400b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 1 --2- 172.21.15.139:0/3914976565 >> v2:172.21.15.139:3300/0 conn(0x7f0540147410 0x7f05400b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 1 monclient: found mon.noname-b 2024-07-26T23:35:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.726+0000 7f0544abd640 1 -- 172.21.15.139:0/3914976565 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054006d700 con 0x7f0540140e80 2024-07-26T23:35:25.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 10 client.4326.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 15 client.4326.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 20 client.4326.objecter _get_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 20 client.4326.objecter put_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.772 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 20 client.4326.objecter _get_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 15 client.4326.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 20 client.4326.objecter get_session s=0x7fa39007e900 osd=0 3 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 15 client.4326.objecter _session_command_op_assign 0 2 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 20 client.4326.objecter put_session s=0x7fa39007e900 osd=0 4 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 10 client.4326.objecter _send_command 2 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.727+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 --> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3b00afb30 con 0x7fa39007efd0 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.728+0000 7f887c85a640 10 client.4335.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-07-26T23:35:25.773 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.728+0000 7f887c85a640 15 client.4335.objecter _session_command_op_assign -1 2 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.728+0000 7f887c85a640 20 client.4335.objecter _get_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.728+0000 7f887c85a640 20 client.4335.objecter put_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.728+0000 7f887c85a640 20 client.4335.objecter _get_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.729+0000 7f887c85a640 15 client.4335.objecter _session_command_op_remove -1 2 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.729+0000 7f887c85a640 20 client.4335.objecter get_session s=0x7f886407e710 osd=1 3 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.729+0000 7f887c85a640 15 client.4335.objecter _session_command_op_assign 1 2 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.729+0000 7f887c85a640 20 client.4335.objecter put_session s=0x7f886407e710 osd=1 4 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.729+0000 7f887c85a640 10 client.4335.objecter _send_command 2 2024-07-26T23:35:25.774 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.729+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88780aed00 con 0x7f886407ede0 2024-07-26T23:35:25.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.730+0000 7f0544abd640 1 --2- 172.21.15.139:0/3914976565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540140e80 0x7f0540146ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f05300098a0 tx=0x7f053002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8149f9a93a25f603 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 1 -- 172.21.15.139:0/3914976565 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530035020 con 0x7f0540140e80 2024-07-26T23:35:25.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: dump: 2024-07-26T23:35:25.777 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.777 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 1 -- 172.21.15.139:0/3914976565 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0530031640 con 0x7f0540140e80 2024-07-26T23:35:25.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 1 -- 172.21.15.139:0/3914976565 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053002d040 con 0x7f0540140e80 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0537fff640 10 monclient: dump: 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.779 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0547522640 5 monclient: authenticate success, global_id 4301 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0547522640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.733+0000 7f0547522640 4 set_mon_vals no callback set 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 10 monclient: shutdown 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 -- 172.21.15.139:0/3914976565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540140e80 msgr2=0x7f0540146ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 --2- 172.21.15.139:0/3914976565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540140e80 0x7f0540146ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f05300098a0 tx=0x7f053002b8f0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 -- 172.21.15.139:0/3914976565 shutdown_connections 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 --2- 172.21.15.139:0/3914976565 >> v2:172.21.15.139:3300/0 conn(0x7f0540147410 0x7f05400b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 --2- 172.21.15.139:0/3914976565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540140e80 0x7f0540146ed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 --2- 172.21.15.139:0/3914976565 >> v2:172.21.15.177:3300/0 conn(0x7f0540068740 0x7f0540140940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 -- 172.21.15.139:0/3914976565 >> 172.21.15.139:0/3914976565 conn(0x7f054006c720 msgr2=0x7f054006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 -- 172.21.15.139:0/3914976565 shutdown_connections 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.734+0000 7f0547522640 1 -- 172.21.15.139:0/3914976565 wait complete. 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.735+0000 7f0547522640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.735+0000 7f0547522640 10 monclient: monmap: 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:25.781 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.735+0000 7f0547522640 1 librados: starting msgr at 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.735+0000 7f0547522640 1 librados: starting objecter 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 Processor -- start 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 -- start start 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 librados: setting wanted keys 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 librados: calling monclient init 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient: init 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0540068740 0x7f0540073d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): picked mon.noname-a con 0x7f0540068740 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540140e80 0x7f0540072030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): picked mon.noname-b con 0x7f0540140e80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540147410 0x7f05401dd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): picked mon.noname-c con 0x7f0540147410 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f054006d9a0 con 0x7f0540068740 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0540141490 con 0x7f0540140e80 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f05400af8f0 con 0x7f0540147410 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.736+0000 7f0547522640 10 monclient(hunting): authenticate will time out at 1358.771953s 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.737+0000 7f0545abf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540147410 0x7f05401dd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.737+0000 7f0545abf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540147410 0x7f05401dd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60778/0 (socket says 172.21.15.139:60778) 2024-07-26T23:35:25.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.737+0000 7f0545abf640 1 -- 172.21.15.139:0/1584814733 learned_addr learned my addr 172.21.15.139:0/1584814733 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.737+0000 7f0545abf640 10 monclient(hunting): get_auth_request con 0x7f0540147410 auth_method 0 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.737+0000 7f0545abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.737+0000 7f0545abf640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.737+0000 7f0545abf640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.738+0000 7f0545abf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.738+0000 7f0545abf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.738+0000 7f0545abf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.740+0000 7f05452be640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0540068740 0x7f0540073d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7fa3a7fff640 1 -- 172.21.15.139:0/2859897221 <== osd.0 v2:172.21.15.139:6808/2123669081 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa3b00afb30 con 0x7fa39007efd0 2024-07-26T23:35:25.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7fa3a7fff640 10 client.4326.objecter ms_dispatch 0x7fa3b0000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7fa3a7fff640 10 client.4326.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7fa3a7fff640 20 client.4326.objecter put_session s=0x7fa39007e900 osd=0 4 2024-07-26T23:35:25.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7fa3a7fff640 15 client.4326.objecter _session_command_op_remove 0 2 2024-07-26T23:35:25.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.740+0000 7f05452be640 10 monclient(hunting): get_auth_request con 0x7f0540068740 auth_method 0 2024-07-26T23:35:25.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.740+0000 7f05452be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.740+0000 7f05452be640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.740+0000 7f05452be640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7f05452be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7f05452be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.741+0000 7f05452be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.787 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-07-26T23:35:25.787 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 1 -- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540140e80 msgr2=0x7f0540072030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.787 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540140e80 0x7f0540072030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 10 librados: watch_flush enter 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 10 librados: watch_flush exit 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 10 client.4326.objecter close_session for osd.0 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7fa39007efd0 msgr2=0x7fa390081410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7fa39007efd0 0x7fa390081410 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 20 client.4326.objecter put_session s=0x7fa39007e900 osd=0 1 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 20 client.4326.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 10 client.4326.objecter successfully canceled tick 2024-07-26T23:35:25.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 1 -- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0540068740 msgr2=0x7f0540073d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0540068740 0x7f0540073d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 1 monclient: found mon.noname-c 2024-07-26T23:35:25.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7f0545abf640 1 -- 172.21.15.139:0/1584814733 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0540072cd0 con 0x7f0540147410 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa390078b90 msgr2=0x7fa39007b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa390078b90 0x7fa39007b050 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0003b80 tx=0x7fa3a00098d0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 10 monclient: shutdown 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa3b00ace50 msgr2=0x7fa3b01e9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 --2- 172.21.15.139:0/2859897221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa3b00ace50 0x7fa3b01e9db0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa39800bc20 tx=0x7fa3980046c0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3affff640 1 -- 172.21.15.139:0/2859897221 reap_dead start 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 shutdown_connections 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.742+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 >> 172.21.15.139:0/2859897221 conn(0x7fa3b00b5870 msgr2=0x7fa3b01522e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 shutdown_connections 2024-07-26T23:35:25.791 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 1 -- 172.21.15.139:0/2746668543 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510017070 con 0x7fa514030880 2024-07-26T23:35:25.791 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.791 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa3b5c29640 1 -- 172.21.15.139:0/2859897221 wait complete. 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: dump: 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.792 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 5 monclient: authenticate success, global_id 4350 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 10 monclient: _renew_subs 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5140c8010 con 0x7fa514030880 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 1 -- 172.21.15.139:0/2746668543 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5100035f0 con 0x7fa514030880 2024-07-26T23:35:25.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 1 -- 172.21.15.139:0/2746668543 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510003a20 con 0x7fa514030880 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa5197fa640 10 monclient: dump: 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.794 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 10 monclient: _renew_subs 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5140c6e10 con 0x7fa514030880 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa521c7b640 1 librados: init done 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa3b5c29640 1 librados: shutdown 2024-07-26T23:35:25.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7f885ffff640 1 -- 172.21.15.139:0/3663648969 <== osd.1 v2:172.21.15.139:6800/3107328002 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f88780aed00 con 0x7f886407ede0 2024-07-26T23:35:25.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7f885ffff640 10 client.4335.objecter ms_dispatch 0x7f8878000b90 command_reply(tid 2: 0 ) 2024-07-26T23:35:25.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7f885ffff640 10 client.4335.objecter _finish_command 2 = system:0 2024-07-26T23:35:25.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7f885ffff640 20 client.4335.objecter put_session s=0x7f886407e710 osd=1 4 2024-07-26T23:35:25.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7f885ffff640 15 client.4335.objecter _session_command_op_remove 1 2 2024-07-26T23:35:25.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.743+0000 7fa51a7fc640 4 set_mon_vals no callback set 2024-07-26T23:35:25.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7fa521c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7fa521c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7fa521c7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:25.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 10 librados: watch_flush enter 2024-07-26T23:35:25.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5140331a0 con 0x7fa514030880 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 10 librados: watch_flush exit 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 10 client.4335.objecter close_session for osd.1 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f886407ede0 msgr2=0x7f8864081220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f886407ede0 0x7f8864081220 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 20 client.4335.objecter put_session s=0x7f886407e710 osd=1 1 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 20 client.4335.objecter shutdown clearing up homeless session... 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.744+0000 7f887c85a640 10 client.4335.objecter successfully canceled tick 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f88640789a0 msgr2=0x7f886407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f88640789a0 0x7f886407ae60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8868006a90 tx=0x7f8868006a20 comp rx=0 tx=0).stop 2024-07-26T23:35:25.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 10 monclient: shutdown 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 msgr2=0x7f8878071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 --2- 172.21.15.139:0/3663648969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f887813c6f0 0x7f8878071880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8870030a20 tx=0x7f8870004a40 comp rx=0 tx=0).stop 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f8877fff640 1 -- 172.21.15.139:0/3663648969 reap_dead start 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 shutdown_connections 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 >> 172.21.15.139:0/3663648969 conn(0x7f88780b1c10 msgr2=0x7f88780b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 shutdown_connections 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 -- 172.21.15.139:0/3663648969 wait complete. 2024-07-26T23:35:25.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f887c85a640 1 librados: shutdown 2024-07-26T23:35:25.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f0545abf640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540147410 0x7f05401dd1b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f053800b9a0 tx=0x7f053800bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f0535ffb640 10 client.?.objecter ms_handle_connect 0x7f0540147410 2024-07-26T23:35:25.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.745+0000 7f0535ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:25.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0535ffb640 1 -- 172.21.15.139:0/1584814733 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538019070 con 0x7f0540147410 2024-07-26T23:35:25.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0535ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0535ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0535ffb640 10 monclient: dump: 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0535ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0535ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0535ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 5 monclient: authenticate success, global_id 4304 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 10 monclient: _renew_subs 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0540072fb0 con 0x7f0540147410 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 10 monclient: _renew_subs 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05401dd6f0 con 0x7f0540147410 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.749+0000 7f0547522640 1 librados: init done 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f0535ffb640 1 -- 172.21.15.139:0/1584814733 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053800ca40 con 0x7f0540147410 2024-07-26T23:35:25.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f0535ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f0535ffb640 1 -- 172.21.15.139:0/1584814733 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538023060 con 0x7f0540147410 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f0535ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f0535ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f0535ffb640 10 monclient: dump: 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.806 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.807 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.807 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f05377fe640 4 set_mon_vals no callback set 2024-07-26T23:35:25.807 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.750+0000 7f0547522640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.807 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.751+0000 7f0547522640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.807 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.751+0000 7f0547522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:25.807 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.751+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0520005080 con 0x7f0540147410 2024-07-26T23:35:25.808 INFO:teuthology.orchestra.run.smithi139.stdout:85899345929 2024-07-26T23:35:25.808 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-07-26T23:35:25.809 INFO:teuthology.orchestra.run.smithi139.stdout:90194313229 2024-07-26T23:35:25.809 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.852+0000 7f2f448cd640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.852+0000 7f2f448cd640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.852+0000 7f2f448cd640 10 monclient: monmap: 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.852494+0000 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.852494+0000 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.853 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.854 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.854 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.854 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.854 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.854 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.853+0000 7f2f448cd640 1 Processor -- start 2024-07-26T23:35:25.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.853+0000 7f2f448cd640 1 -- start start 2024-07-26T23:35:25.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.853+0000 7f2f448cd640 10 monclient: init 2024-07-26T23:35:25.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.855 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f401470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2f400b41c0 0x7f2f400b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient(hunting): picked mon.noname-c con 0x7f2f400b41c0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f4014c1d0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f40147ce0 con 0x7f2f401470c0 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f400acaf0 con 0x7f2f4014c1d0 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2f401477c0 con 0x7f2f400b41c0 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.854+0000 7f2f448cd640 10 monclient(hunting): authenticate will time out at 1358.889325s 2024-07-26T23:35:25.889 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.889+0000 7f0535ffb640 1 -- 172.21.15.139:0/1584814733 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f053801c070 con 0x7f0540147410 2024-07-26T23:35:25.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.889+0000 7f0535ffb640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f05240789a0 0x7f052407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.890+0000 7f05452be640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f05240789a0 0x7f052407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.890+0000 7f05452be640 10 monclient: get_auth_request con 0x7f05240789a0 auth_method 0 2024-07-26T23:35:25.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.890+0000 7f0535ffb640 10 client.4304.objecter ms_handle_connect 0x7f05240789a0 2024-07-26T23:35:25.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.890+0000 7f05452be640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f05240789a0 0x7f052407ae60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f053c009990 tx=0x7f053c009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.891+0000 7f0ef3591640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.891+0000 7f0ef3591640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.891+0000 7f0ef3591640 10 monclient: monmap: 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.891927+0000 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.891927+0000 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.892 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.893 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.893 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.893 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.893 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.892+0000 7f0ef3591640 1 Processor -- start 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.892+0000 7f0ef3591640 1 -- start start 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.892+0000 7f0ef3591640 10 monclient: init 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0eec1470c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0eec0b41c0 0x7f0eec0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0535ffb640 1 -- 172.21.15.139:0/1584814733 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f053800cdf0 con 0x7f0540147410 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0535ffb640 10 client.4304.objecter ms_dispatch 0x7f0540000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0535ffb640 3 client.4304.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0535ffb640 3 client.4304.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0535ffb640 20 client.4304.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient(hunting): picked mon.noname-c con 0x7f0eec0b41c0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0eec14c1d0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eec147ce0 con 0x7f0eec14c1d0 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eec0acaf0 con 0x7f0eec1470c0 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0eec1477c0 con 0x7f0eec0b41c0 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.893+0000 7f0ef3591640 10 monclient(hunting): authenticate will time out at 1358.928613s 2024-07-26T23:35:25.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.899+0000 7fa5197fa640 1 -- 172.21.15.139:0/2746668543 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fa51001a070 con 0x7fa514030880 2024-07-26T23:35:25.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.899+0000 7fa5197fa640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa500078a20 0x7fa50007aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.899+0000 7fa51bfff640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa500078a20 0x7fa50007aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.899+0000 7fa51bfff640 10 monclient: get_auth_request con 0x7fa500078a20 auth_method 0 2024-07-26T23:35:25.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.900+0000 7fa51bfff640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa500078a20 0x7fa50007aee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa508036000 tx=0x7fa508030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.901 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.900+0000 7fa5197fa640 10 client.4350.objecter ms_handle_connect 0x7fa500078a20 2024-07-26T23:35:25.905 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.905+0000 7fa5197fa640 1 -- 172.21.15.139:0/2746668543 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fa51005cdf0 con 0x7fa514030880 2024-07-26T23:35:25.905 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.905+0000 7fa5197fa640 10 client.4350.objecter ms_dispatch 0x7fa51c000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.905 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.905+0000 7fa5197fa640 3 client.4350.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.905 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.905+0000 7fa5197fa640 3 client.4350.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.905 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.905+0000 7fa5197fa640 20 client.4350.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.940+0000 7f2a394b5640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.940+0000 7f2a394b5640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.940+0000 7f2a394b5640 10 monclient: monmap: 2024-07-26T23:35:25.941 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.941 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.941 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.940944+0000 2024-07-26T23:35:25.941 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.940944+0000 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.942 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.941+0000 7f2a394b5640 1 Processor -- start 2024-07-26T23:35:25.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.941+0000 7f2a394b5640 1 -- start start 2024-07-26T23:35:25.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.941+0000 7f2a394b5640 10 monclient: init 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a3406e720 0x7f2a34139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a3406e720 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a34143d00 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a34144cd0 0x7f2a34145120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a34144cd0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a340ac810 con 0x7f2a34143d00 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2a3406eb00 con 0x7f2a34144cd0 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f2a34144710 con 0x7f2a3406e720 2024-07-26T23:35:25.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.945 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.942+0000 7f2a394b5640 10 monclient(hunting): authenticate will time out at 1358.977502s 2024-07-26T23:35:25.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.959+0000 7f16017fa640 1 -- 172.21.15.139:0/1013571710 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f15f8037070 con 0x7f16040b48d0 2024-07-26T23:35:25.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.959+0000 7f16017fa640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f15e80789d0 0x7f15e807ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.959+0000 7f1603fff640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f15e80789d0 0x7f15e807ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.960+0000 7f1603fff640 10 monclient: get_auth_request con 0x7f15e80789d0 auth_method 0 2024-07-26T23:35:25.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.960+0000 7f16017fa640 10 client.4345.objecter ms_handle_connect 0x7f15e80789d0 2024-07-26T23:35:25.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.960+0000 7f1603fff640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f15e80789d0 0x7f15e807ae90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16041d68a0 tx=0x7f15f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.962+0000 7f16017fa640 1 -- 172.21.15.139:0/1013571710 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f15f80305c0 con 0x7f16040b48d0 2024-07-26T23:35:25.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.962+0000 7f16017fa640 10 client.4345.objecter ms_dispatch 0x7f1604000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:25.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.962+0000 7f16017fa640 3 client.4345.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:25.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.962+0000 7f16017fa640 3 client.4345.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:25.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.963+0000 7f16017fa640 20 client.4345.objecter dump_active .. 0 homeless 2024-07-26T23:35:25.967 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.967+0000 7f6f1e92d640 10 monclient: get_monmap_and_config 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.967+0000 7f6f1e92d640 10 monclient: build_initial_monmap 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.967+0000 7f6f1e92d640 10 monclient: monmap: 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:25.967758+0000 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:25.967758+0000 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:25.968 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:25.969 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.968+0000 7f6f1e92d640 1 Processor -- start 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.968+0000 7f6f1e92d640 1 -- start start 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.968+0000 7f6f1e92d640 10 monclient: init 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f181470c0 0x7f6f18140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f181470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f18140e90 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6f180b48d0 0x7f6f1814e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f180b48d0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f18147ce0 con 0x7f6f18140e90 2024-07-26T23:35:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f6f180acaf0 con 0x7f6f180b48d0 2024-07-26T23:35:25.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:25.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f6f181477c0 con 0x7f6f181470c0 2024-07-26T23:35:25.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:25.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.969+0000 7f6f1e92d640 10 monclient(hunting): authenticate will time out at 1359.004183s 2024-07-26T23:35:25.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.978+0000 7f16017fa640 1 -- 172.21.15.139:0/1013571710 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f15f8083b90 con 0x7f16040b48d0 2024-07-26T23:35:25.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.978+0000 7f16017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.978+0000 7f16017fa640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:25.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 1 -- 172.21.15.139:0/2846045856 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e900125f0 con 0x7f3e9c13ce20 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: dump: 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.982 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 1 -- 172.21.15.139:0/2846045856 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e90018070 con 0x7f3e9c13ce20 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 1 -- 172.21.15.139:0/2846045856 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e900129d0 con 0x7f3e9c13ce20 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e83fff640 10 monclient: dump: 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:25.983 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 5 monclient: authenticate success, global_id 4348 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 10 monclient: _renew_subs 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3e99ffb640 4 set_mon_vals no callback set 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9c13f630 con 0x7f3e9c13ce20 2024-07-26T23:35:25.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:25.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f2f3dd9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2f400b41c0 0x7f2f400b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 10 monclient: _renew_subs 2024-07-26T23:35:25.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9c139920 con 0x7f3e9c13ce20 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.981+0000 7f3ea08b3640 1 librados: init done 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f3ea08b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f3ea08b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f3ea08b3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e9c0aeda0 con 0x7f3e9c13ce20 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f2f3dd9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2f400b41c0 0x7f2f400b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41356/0 (socket says 172.21.15.139:41356) 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/1231919477 learned_addr learned my addr 172.21.15.139:0/1231919477 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f2f3dd9b640 10 monclient(hunting): get_auth_request con 0x7f2f400b41c0 auth_method 0 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f2f3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f2f3dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f2f3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.982+0000 7f6f1ceca640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6f180b48d0 0x7f6f1814e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.983+0000 7f6f1ceca640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6f180b48d0 0x7f6f1814e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41388/0 (socket says 172.21.15.139:41388) 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.983+0000 7f6f1ceca640 1 -- 172.21.15.139:0/3031237397 learned_addr learned my addr 172.21.15.139:0/3031237397 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.983+0000 7f6f1ceca640 10 monclient(hunting): get_auth_request con 0x7f6f180b48d0 auth_method 0 2024-07-26T23:35:25.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.983+0000 7f6f1ceca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.983+0000 7f6f1ceca640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.983+0000 7f6f1ceca640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.984+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.984+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.984+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.984+0000 7f2a337fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a34144cd0 0x7f2a34145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f2a337fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a34144cd0 0x7f2a34145120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41372/0 (socket says 172.21.15.139:41372) 2024-07-26T23:35:25.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f2a337fe640 1 -- 172.21.15.139:0/1958887632 learned_addr learned my addr 172.21.15.139:0/1958887632 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f2a337fe640 10 monclient(hunting): get_auth_request con 0x7f2a34144cd0 auth_method 0 2024-07-26T23:35:25.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f2a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f2a337fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f2a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f6f1ceca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f6f1ceca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.985+0000 7f6f1ceca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2024-07-26T23:35:25.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/1231919477 >> v1:172.21.15.177:6789/0 conn(0x7f2f4014c1d0 legacy=0x7f2f4014e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/1231919477 >> v1:172.21.15.139:6789/0 conn(0x7f2f401470c0 legacy=0x7f2f40140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 1 monclient: found mon.noname-c 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/1231919477 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f401474a0 con 0x7f2f400b41c0 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.987+0000 7f2a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f0ef0b2c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0eec0b41c0 0x7f0eec0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:25.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-07-26T23:35:25.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 1 -- 172.21.15.139:0/3031237397 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f181470c0 msgr2=0x7f6f18140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 1 --2- 172.21.15.139:0/3031237397 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f181470c0 0x7f6f18140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:25.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 1 -- 172.21.15.139:0/3031237397 >> v1:172.21.15.177:6789/0 conn(0x7f6f18140e90 legacy=0x7f6f180b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:25.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 1 monclient: found mon.noname-c 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f6f1ceca640 1 -- 172.21.15.139:0/3031237397 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f181474a0 con 0x7f6f180b48d0 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f0ef0b2c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0eec0b41c0 0x7f0eec0b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41362/0 (socket says 172.21.15.139:41362) 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f0ef0b2c640 1 -- 172.21.15.139:0/2885576121 learned_addr learned my addr 172.21.15.139:0/2885576121 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f0ef0b2c640 10 monclient(hunting): get_auth_request con 0x7f0eec0b41c0 auth_method 0 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f0ef0b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f0ef0b2c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:25.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.991+0000 7f0ef0b2c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:25.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.995+0000 7f6f1ceca640 1 --2- 172.21.15.139:0/3031237397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f180b48d0 0x7f6f1814e1e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009e50 tx=0x7f6f1002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a915af88f93df6c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:25.996 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.996+0000 7f0ef0b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:25.996 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.996+0000 7f0ef0b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:25.997 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.996+0000 7f0ef0b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:25.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2024-07-26T23:35:25.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 1 -- 172.21.15.139:0/2885576121 >> v1:172.21.15.177:6789/0 conn(0x7f0eec1470c0 legacy=0x7f0eec140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 1 -- 172.21.15.139:0/2885576121 >> v1:172.21.15.139:6789/0 conn(0x7f0eec14c1d0 legacy=0x7f0eec14e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:25.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:25.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 1 monclient: found mon.noname-c 2024-07-26T23:35:26.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:26.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:26.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:25.999+0000 7f0ef0b2c640 1 -- 172.21.15.139:0/2885576121 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec1474a0 con 0x7f0eec0b41c0 2024-07-26T23:35:26.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2024-07-26T23:35:26.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 1 -- 172.21.15.139:0/1958887632 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a3406e720 msgr2=0x7f2a34139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:26.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 1 --2- 172.21.15.139:0/1958887632 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a3406e720 0x7f2a34139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 1 -- 172.21.15.139:0/1958887632 >> v1:172.21.15.139:6789/0 conn(0x7f2a34143d00 legacy=0x7f2a3413a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:26.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:26.003 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 1 monclient: found mon.noname-c 2024-07-26T23:35:26.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:26.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:26.004 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.003+0000 7f2a337fe640 1 -- 172.21.15.139:0/1958887632 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a340acaf0 con 0x7f2a34144cd0 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 1 -- 172.21.15.139:0/3031237397 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1002cb00 con 0x7f6f180b48d0 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: dump: 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.007 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 1 -- 172.21.15.139:0/3031237397 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f1002cca0 con 0x7f6f180b48d0 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 1 -- 172.21.15.139:0/3031237397 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10034610 con 0x7f6f180b48d0 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f1e92d640 5 monclient: authenticate success, global_id 4354 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f1e92d640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f1e92d640 4 set_mon_vals no callback set 2024-07-26T23:35:26.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.006+0000 7f6f16ffd640 10 monclient: dump: 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.009 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 10 monclient: shutdown 2024-07-26T23:35:26.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 -- 172.21.15.139:0/3031237397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f180b48d0 msgr2=0x7f6f1814e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/3031237397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f180b48d0 0x7f6f1814e1e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009e50 tx=0x7f6f1002bf60 comp rx=0 tx=0).stop 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 -- 172.21.15.139:0/3031237397 shutdown_connections 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/3031237397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f180b48d0 0x7f6f1814e1e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/3031237397 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f181470c0 0x7f6f18140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 -- 172.21.15.139:0/3031237397 >> 172.21.15.139:0/3031237397 conn(0x7f6f180b62c0 msgr2=0x7f6f180b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 -- 172.21.15.139:0/3031237397 shutdown_connections 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.007+0000 7f6f1e92d640 1 -- 172.21.15.139:0/3031237397 wait complete. 2024-07-26T23:35:26.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.008+0000 7f0ef0b2c640 1 --2- 172.21.15.139:0/2885576121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec0b41c0 0x7f0eec0b45e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdcb082ce1e2c797 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.008+0000 7f6f1e92d640 10 monclient: build_initial_monmap 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.008+0000 7f6f1e92d640 10 monclient: monmap: 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:26.012 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:26.013 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:26.013 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.008+0000 7f6f1e92d640 1 librados: starting msgr at 2024-07-26T23:35:26.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.008+0000 7f6f1e92d640 1 librados: starting objecter 2024-07-26T23:35:26.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.008+0000 7f2a337fe640 1 --2- 172.21.15.139:0/1958887632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34144cd0 0x7f2a34145120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009e50 tx=0x7f2a2c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=877e231eb67118d7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.008+0000 7f6f1e92d640 1 Processor -- start 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 -- start start 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 librados: setting wanted keys 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 librados: calling monclient init 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient: init 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f180b48d0 0x7f6f18145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f180b48d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f18140e90 0x7f6f18143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f18140e90 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f181470c0 0x7f6f181358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f181470c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6f18141410 con 0x7f6f18140e90 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f18074850 con 0x7f6f181470c0 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1814e720 con 0x7f6f180b48d0 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.009+0000 7f6f1e92d640 10 monclient(hunting): authenticate will time out at 1359.044757s 2024-07-26T23:35:26.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.014+0000 7f6f1ceca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f181470c0 0x7f6f181358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2f3dd9b640 1 --2- 172.21.15.139:0/1231919477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f400b41c0 0x7f2f400b45e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009870 tx=0x7f2f3402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c589086038eb776 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a31ffb640 1 -- 172.21.15.139:0/1958887632 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c02caf0 con 0x7f2a34144cd0 2024-07-26T23:35:26.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:26.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a31ffb640 10 monclient: dump: 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a31ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a31ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a394b5640 5 monclient: authenticate success, global_id 4357 2024-07-26T23:35:26.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f2a394b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:26.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f6f1ceca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f181470c0 0x7f6f181358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41398/0 (socket says 172.21.15.139:41398) 2024-07-26T23:35:26.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f6f1ceca640 1 -- 172.21.15.139:0/4171605067 learned_addr learned my addr 172.21.15.139:0/4171605067 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:26.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f6f1ceca640 10 monclient(hunting): get_auth_request con 0x7f6f181470c0 auth_method 0 2024-07-26T23:35:26.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f6f1ceca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:26.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f6f1ceca640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:26.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.015+0000 7f6f1ceca640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:26.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a31ffb640 1 -- 172.21.15.139:0/1958887632 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c02cc90 con 0x7f2a34144cd0 2024-07-26T23:35:26.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a31ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a31ffb640 1 -- 172.21.15.139:0/1958887632 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c034610 con 0x7f2a34144cd0 2024-07-26T23:35:26.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a394b5640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:26.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a394b5640 4 set_mon_vals no callback set 2024-07-26T23:35:26.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.020 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.017+0000 7f2a31ffb640 10 monclient: dump: 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 10 monclient: shutdown 2024-07-26T23:35:26.021 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.022 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 -- 172.21.15.139:0/1958887632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34144cd0 msgr2=0x7f2a34145120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.022 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f6f1ceca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:26.022 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f6f1ceca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:26.022 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f6f1ceca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:26.023 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 --2- 172.21.15.139:0/1958887632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34144cd0 0x7f2a34145120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009e50 tx=0x7f2a2c02bf50 comp rx=0 tx=0).stop 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 -- 172.21.15.139:0/1958887632 shutdown_connections 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 --2- 172.21.15.139:0/1958887632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34144cd0 0x7f2a34145120 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 --2- 172.21.15.139:0/1958887632 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a3406e720 0x7f2a34139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 -- 172.21.15.139:0/1958887632 >> 172.21.15.139:0/1958887632 conn(0x7f2a340b9c10 msgr2=0x7f2a340ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 -- 172.21.15.139:0/1958887632 shutdown_connections 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.018+0000 7f2a394b5640 1 -- 172.21.15.139:0/1958887632 wait complete. 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 10 monclient: build_initial_monmap 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 10 monclient: monmap: 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:26.024 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 1 librados: starting msgr at 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 1 librados: starting objecter 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 1 Processor -- start 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 1 -- start start 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 1 librados: setting wanted keys 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 1 librados: calling monclient init 2024-07-26T23:35:26.025 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.019+0000 7f2a394b5640 10 monclient: init 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3406e720 0x7f2a3413f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a3406e720 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34143d00 0x7f2a3413d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a34143d00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a34144cd0 0x7f2a341dcc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a34144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2a34145660 con 0x7f2a3406e720 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a34141390 con 0x7f2a34143d00 2024-07-26T23:35:26.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f2a341457e0 con 0x7f2a34144cd0 2024-07-26T23:35:26.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:26.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.020+0000 7f2a394b5640 10 monclient(hunting): authenticate will time out at 1359.055420s 2024-07-26T23:35:26.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.049+0000 7fa5197fa640 1 -- 172.21.15.139:0/2746668543 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fa510021020 con 0x7fa514030880 2024-07-26T23:35:26.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.049+0000 7fa5197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.050 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.049+0000 7fa5197fa640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:26.051 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.050+0000 7f2a32ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3406e720 0x7f2a3413f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.053 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f6f177fe640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f18140e90 0x7f6f18143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.053 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f2a32ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3406e720 0x7f2a3413f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:32818/0 (socket says 172.21.15.139:32818) 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f2a32ffd640 1 -- 172.21.15.139:0/3886977699 learned_addr learned my addr 172.21.15.139:0/3886977699 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f2a32ffd640 10 monclient(hunting): get_auth_request con 0x7f2a3406e720 auth_method 0 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f2a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f2a32ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f2a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f6f177fe640 10 monclient(hunting): get_auth_request con 0x7f6f18140e90 auth_method 0 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f6f177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f6f177fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:26.054 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.053+0000 7f6f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:26.055 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.054+0000 7f2a32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:26.055 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.054+0000 7f2a32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:26.055 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.054+0000 7f2a32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:26.055 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.055+0000 7f6f177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:26.055 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.055+0000 7f6f177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:26.056 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.055+0000 7f6f177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:26.058 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-07-26T23:35:26.058 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 1 -- 172.21.15.139:0/3886977699 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a34144cd0 msgr2=0x7f2a341dcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.058 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a34144cd0 0x7f2a341dcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.058 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 1 -- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34143d00 msgr2=0x7f2a3413d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34143d00 0x7f2a3413d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:26.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 1 monclient: found mon.noname-a 2024-07-26T23:35:26.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:26.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.059 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.058+0000 7f2a32ffd640 1 -- 172.21.15.139:0/3886977699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3413cd80 con 0x7f2a3406e720 2024-07-26T23:35:26.063 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-07-26T23:35:26.063 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 1 -- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f180b48d0 msgr2=0x7f6f18145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f180b48d0 0x7f6f18145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 1 -- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f181470c0 msgr2=0x7f6f181358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f181470c0 0x7f6f181358a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 1 monclient: found mon.noname-a 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.063+0000 7f6f177fe640 1 -- 172.21.15.139:0/4171605067 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f18143bd0 con 0x7f6f18140e90 2024-07-26T23:35:26.067 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.067+0000 7f2a32ffd640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3406e720 0x7f2a3413f9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400b4a0 tx=0x7f2a2400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.067 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.067+0000 7f2a1b7fe640 10 client.?.objecter ms_handle_connect 0x7f2a3406e720 2024-07-26T23:35:26.067 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.067+0000 7f2a1b7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:26.068 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.068+0000 7f6f177fe640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f18140e90 0x7f6f18143690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00c3c0 tx=0x7f6f0c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.068 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.068+0000 7f6f14ff9640 10 client.?.objecter ms_handle_connect 0x7f6f18140e90 2024-07-26T23:35:26.068 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.068+0000 7f6f14ff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:26.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a1b7fe640 1 -- 172.21.15.139:0/3886977699 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24014070 con 0x7f2a3406e720 2024-07-26T23:35:26.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:26.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a1b7fe640 10 monclient: dump: 2024-07-26T23:35:26.072 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.072 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.072 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a1b7fe640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a1b7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 5 monclient: authenticate success, global_id 4356 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 10 monclient: _renew_subs 2024-07-26T23:35:26.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3413d2d0 con 0x7f2a3406e720 2024-07-26T23:35:26.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:26.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 10 monclient: _renew_subs 2024-07-26T23:35:26.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a34139bb0 con 0x7f2a3406e720 2024-07-26T23:35:26.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.071+0000 7f2a394b5640 1 librados: init done 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.073+0000 7f2a394b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.073+0000 7f2a394b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.073+0000 7f2a394b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.073+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a340ac250 con 0x7f2a3406e720 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.074+0000 7f2a1b7fe640 1 -- 172.21.15.139:0/3886977699 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a24010610 con 0x7f2a3406e720 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.074+0000 7f2a1b7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.074+0000 7f2a1b7fe640 1 -- 172.21.15.139:0/3886977699 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2401e060 con 0x7f2a3406e720 2024-07-26T23:35:26.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.074+0000 7f2a1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.074+0000 7f2a1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.074+0000 7f2a1b7fe640 10 monclient: dump: 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.076 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.074+0000 7f2a317fa640 4 set_mon_vals no callback set 2024-07-26T23:35:26.078 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 1 -- 172.21.15.139:0/4171605067 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c017070 con 0x7f6f18140e90 2024-07-26T23:35:26.078 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.078 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: dump: 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.079 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 1 -- 172.21.15.139:0/4171605067 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f0c0052d0 con 0x7f6f18140e90 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 5 monclient: authenticate success, global_id 4359 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 10 monclient: _renew_subs 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f18142df0 con 0x7f6f18140e90 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 1 -- 172.21.15.139:0/4171605067 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c003830 con 0x7f6f18140e90 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.080 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:26.081 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f14ff9640 10 monclient: dump: 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f167fc640 4 set_mon_vals no callback set 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:26.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 10 monclient: _renew_subs 2024-07-26T23:35:26.083 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.083 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f18135fc0 con 0x7f6f18140e90 2024-07-26T23:35:26.083 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.078+0000 7f6f1e92d640 1 librados: init done 2024-07-26T23:35:26.083 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.079+0000 7f6f1e92d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.083 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.079+0000 7f6f1e92d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.083 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.079+0000 7f6f1e92d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.083 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.079+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f18143e00 con 0x7f6f18140e90 2024-07-26T23:35:26.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.099+0000 7f2a1b7fe640 1 -- 172.21.15.139:0/3886977699 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f2a24017070 con 0x7f2a3406e720 2024-07-26T23:35:26.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.100+0000 7f2a1b7fe640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2a200789d0 0x7f2a2007ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.100+0000 7f2a327fc640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2a200789d0 0x7f2a2007ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.100+0000 7f2a327fc640 10 monclient: get_auth_request con 0x7f2a200789d0 auth_method 0 2024-07-26T23:35:26.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.100+0000 7f2a1b7fe640 10 client.4356.objecter ms_handle_connect 0x7f2a200789d0 2024-07-26T23:35:26.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.100+0000 7f2a327fc640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2a200789d0 0x7f2a2007ae90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2a3413f750 tx=0x7f2a28007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.151+0000 7f0535ffb640 1 -- 172.21.15.139:0/1584814733 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f0538061ae0 con 0x7f0540147410 2024-07-26T23:35:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.151+0000 7f0535ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.151+0000 7f0535ffb640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:26.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.225+0000 7f160b154640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-26T23:35:26.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.225+0000 7f160b154640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-26T23:35:26.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.225+0000 7f160b154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:26.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.225+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f160406b560 con 0x7f16040b48d0 2024-07-26T23:35:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 1 -- 172.21.15.139:0/2885576121 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc0317c0 con 0x7f0eec0b41c0 2024-07-26T23:35:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: dump: 2024-07-26T23:35:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 1 -- 172.21.15.139:0/2885576121 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0edc037070 con 0x7f0eec0b41c0 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ef3591640 5 monclient: authenticate success, global_id 4363 2024-07-26T23:35:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ef3591640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ef3591640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ef3591640 4 set_mon_vals no callback set 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 1 -- 172.21.15.139:0/2885576121 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc031b80 con 0x7f0eec0b41c0 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.266+0000 7f0ee3fff640 10 monclient: dump: 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.270 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.270 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.270 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 10 monclient: shutdown 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 1 -- 172.21.15.139:0/2885576121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec0b41c0 msgr2=0x7f0eec0b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 1 --2- 172.21.15.139:0/2885576121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec0b41c0 0x7f0eec0b45e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc02b860 comp rx=0 tx=0).stop 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 1 -- 172.21.15.139:0/2885576121 shutdown_connections 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 1 --2- 172.21.15.139:0/2885576121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec0b41c0 0x7f0eec0b45e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 1 -- 172.21.15.139:0/2885576121 >> 172.21.15.139:0/2885576121 conn(0x7f0eec0b62c0 msgr2=0x7f0eec0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 1 -- 172.21.15.139:0/2885576121 shutdown_connections 2024-07-26T23:35:26.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.267+0000 7f0ef3591640 1 -- 172.21.15.139:0/2885576121 wait complete. 2024-07-26T23:35:26.272 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 1 -- 172.21.15.139:0/1231919477 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f340317c0 con 0x7f2f400b41c0 2024-07-26T23:35:26.272 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.272 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:26.272 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: dump: 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.273 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 1 -- 172.21.15.139:0/1231919477 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f34037070 con 0x7f2f400b41c0 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 1 -- 172.21.15.139:0/1231919477 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34031b80 con 0x7f2f400b41c0 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f448cd640 5 monclient: authenticate success, global_id 4351 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f448cd640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f448cd640 4 set_mon_vals no callback set 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f2f3d59a640 10 monclient: dump: 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.274 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.275 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.275 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.275 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.275 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.275 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.275 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.275 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f0ef3591640 10 monclient: build_initial_monmap 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.268+0000 7f0ef3591640 10 monclient: monmap: 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:26.276 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 10 monclient: shutdown 2024-07-26T23:35:26.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 1 -- 172.21.15.139:0/1231919477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f400b41c0 msgr2=0x7f2f400b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 1 --2- 172.21.15.139:0/1231919477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f400b41c0 0x7f2f400b45e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009870 tx=0x7f2f3402b860 comp rx=0 tx=0).stop 2024-07-26T23:35:26.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 1 -- 172.21.15.139:0/1231919477 shutdown_connections 2024-07-26T23:35:26.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 1 --2- 172.21.15.139:0/1231919477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f400b41c0 0x7f2f400b45e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.277 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 1 -- 172.21.15.139:0/1231919477 >> 172.21.15.139:0/1231919477 conn(0x7f2f400b62c0 msgr2=0x7f2f400b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.278 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f0ef3591640 1 librados: starting msgr at 2024-07-26T23:35:26.278 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f0ef3591640 1 librados: starting objecter 2024-07-26T23:35:26.278 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 1 -- 172.21.15.139:0/1231919477 shutdown_connections 2024-07-26T23:35:26.278 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f2f448cd640 1 -- 172.21.15.139:0/1231919477 wait complete. 2024-07-26T23:35:26.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f0ef3591640 1 Processor -- start 2024-07-26T23:35:26.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f0ef3591640 1 -- start start 2024-07-26T23:35:26.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f0ef3591640 1 librados: setting wanted keys 2024-07-26T23:35:26.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f0ef3591640 1 librados: calling monclient init 2024-07-26T23:35:26.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.269+0000 7f0ef3591640 10 monclient: init 2024-07-26T23:35:26.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:26.279 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f2f448cd640 10 monclient: build_initial_monmap 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0eec0b41c0 0x7f0eec1dc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f2f448cd640 10 monclient: monmap: 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:26.280 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:26.281 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.281 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:26.281 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:26.281 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:26.281 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.282 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient(hunting): picked mon.noname-c con 0x7f0eec0b41c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:26.282 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec1470c0 0x7f0eec145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.282 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient(hunting): picked mon.noname-b con 0x7f0eec1470c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:26.282 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0eec14c1d0 0x7f0eec138c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient(hunting): picked mon.noname-a con 0x7f0eec14c1d0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec141410 con 0x7f0eec14c1d0 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec074850 con 0x7f0eec1470c0 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f0eec0b4b20 con 0x7f0eec0b41c0 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.270+0000 7f0ef3591640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:26.283 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef3591640 10 monclient(hunting): authenticate will time out at 1359.306023s 2024-07-26T23:35:26.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f2f448cd640 1 librados: starting msgr at 2024-07-26T23:35:26.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f2f448cd640 1 librados: starting objecter 2024-07-26T23:35:26.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f2f448cd640 1 Processor -- start 2024-07-26T23:35:26.284 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef132d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0eec0b41c0 0x7f0eec1dc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f2f448cd640 1 -- start start 2024-07-26T23:35:26.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f2f448cd640 1 librados: setting wanted keys 2024-07-26T23:35:26.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f2f448cd640 1 librados: calling monclient init 2024-07-26T23:35:26.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f2f448cd640 10 monclient: init 2024-07-26T23:35:26.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef132d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0eec0b41c0 0x7f0eec1dc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60834/0 (socket says 172.21.15.139:60834) 2024-07-26T23:35:26.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef132d640 1 -- 172.21.15.139:0/724264485 learned_addr learned my addr 172.21.15.139:0/724264485 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:26.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef132d640 10 monclient(hunting): get_auth_request con 0x7f0eec0b41c0 auth_method 0 2024-07-26T23:35:26.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef132d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:26.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef132d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:26.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.271+0000 7f0ef132d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:26.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f0ef132d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:26.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f0ef132d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:26.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f0ef132d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:26.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f400b41c0 0x7f2f401dd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 10 monclient(hunting): picked mon.noname-a con 0x7f2f400b41c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2f401470c0 0x7f2f401dc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 10 monclient(hunting): picked mon.noname-c con 0x7f2f401470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f4014c1d0 0x7f2f40139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4014c1d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.272+0000 7f2f448cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.273+0000 7f2f448cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2f40141410 con 0x7f2f400b41c0 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.273+0000 7f2f448cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.273+0000 7f2f448cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2f40074850 con 0x7f2f4014c1d0 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.273+0000 7f2f448cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:26.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.273+0000 7f2f448cd640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f2f400b4b20 con 0x7f2f401470c0 2024-07-26T23:35:26.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.273+0000 7f2f448cd640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:26.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.273+0000 7f2f448cd640 10 monclient(hunting): authenticate will time out at 1359.308171s 2024-07-26T23:35:26.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-07-26T23:35:26.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 1 -- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec1470c0 msgr2=0x7f0eec145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec1470c0 0x7f0eec145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 1 -- 172.21.15.139:0/724264485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0eec14c1d0 msgr2=0x7f0eec138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0eec14c1d0 0x7f0eec138c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 10 monclient: _finish_hunting 0 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 1 monclient: found mon.noname-c 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.275+0000 7f0ef132d640 1 -- 172.21.15.139:0/724264485 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec1dcea0 con 0x7f0eec0b41c0 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.278+0000 7f2f3dd9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2f401470c0 0x7f2f401dc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.278+0000 7f0ef132d640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0eec0b41c0 0x7f0eec1dc960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800c430 tx=0x7f0ee800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.278+0000 7f0ee1ffb640 10 client.?.objecter ms_handle_connect 0x7f0eec0b41c0 2024-07-26T23:35:26.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.278+0000 7f0ee1ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:26.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.279+0000 7f2f3dd9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2f401470c0 0x7f2f401dc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60844/0 (socket says 172.21.15.139:60844) 2024-07-26T23:35:26.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.279+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/2201736133 learned_addr learned my addr 172.21.15.139:0/2201736133 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:26.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.279+0000 7f2f3dd9b640 10 monclient(hunting): get_auth_request con 0x7f2f401470c0 auth_method 0 2024-07-26T23:35:26.291 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.279+0000 7f2f3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:26.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.279+0000 7f2f3dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:26.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.279+0000 7f2f3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:26.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.281+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:26.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.281+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:26.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.281+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.283+0000 7f3e83fff640 1 -- 172.21.15.139:0/2846045856 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f3e90012b70 con 0x7f3e9c13ce20 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.283+0000 7f3e83fff640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f3e880789d0 0x7f3e8807ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.284+0000 7f3e9b7fe640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f3e880789d0 0x7f3e8807ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.284+0000 7f3e9b7fe640 10 monclient: get_auth_request con 0x7f3e880789d0 auth_method 0 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.284+0000 7f3e9b7fe640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f3e880789d0 0x7f3e8807ae90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c036000 tx=0x7f3e8c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.284+0000 7f3e83fff640 10 client.4348.objecter ms_handle_connect 0x7f3e880789d0 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.286+0000 7f3e83fff640 1 -- 172.21.15.139:0/2846045856 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f3e90012e60 con 0x7f3e9c13ce20 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.286+0000 7f3e83fff640 10 client.4348.objecter ms_dispatch 0x7f3e9c000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:26.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.286+0000 7f3e83fff640 3 client.4348.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:26.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.286+0000 7f3e83fff640 3 client.4348.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:26.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.286+0000 7f3e83fff640 20 client.4348.objecter dump_active .. 0 homeless 2024-07-26T23:35:26.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.288+0000 7f2f3dd9b640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-07-26T23:35:26.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.288+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f4014c1d0 msgr2=0x7f2f40139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.288+0000 7f2f3dd9b640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f4014c1d0 0x7f2f40139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.289+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/2201736133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f400b41c0 msgr2=0x7f2f401dd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.289+0000 7f2f3dd9b640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f400b41c0 0x7f2f401dd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.289+0000 7f2f3dd9b640 10 monclient: _finish_hunting 0 2024-07-26T23:35:26.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.289+0000 7f2f3dd9b640 1 monclient: found mon.noname-c 2024-07-26T23:35:26.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.289+0000 7f2f3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:26.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.289+0000 7f2f3dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.289+0000 7f2f3dd9b640 1 -- 172.21.15.139:0/2201736133 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4013ba80 con 0x7f2f401470c0 2024-07-26T23:35:26.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 1 -- 172.21.15.139:0/724264485 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8011790 con 0x7f0eec0b41c0 2024-07-26T23:35:26.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:26.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: dump: 2024-07-26T23:35:26.296 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.296 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 1 -- 172.21.15.139:0/724264485 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee8017070 con 0x7f0eec0b41c0 2024-07-26T23:35:26.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 5 monclient: authenticate success, global_id 4307 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 10 monclient: _renew_subs 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eec14adc0 con 0x7f0eec0b41c0 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 1 -- 172.21.15.139:0/724264485 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8011b70 con 0x7f0eec0b41c0 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee1ffb640 10 monclient: dump: 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.298 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ee37fe640 4 set_mon_vals no callback set 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 10 monclient: _renew_subs 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eec147fa0 con 0x7f0eec0b41c0 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.291+0000 7f0ef3591640 1 librados: init done 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.292+0000 7f0ef3591640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.292+0000 7f0ef3591640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.292+0000 7f0ef3591640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.292+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eec14ab30 con 0x7f0eec0b41c0 2024-07-26T23:35:26.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.294+0000 7f2f3dd9b640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2f401470c0 0x7f2f401dc450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2f340090b0 tx=0x7f2f340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.294+0000 7f2f2effd640 10 client.?.objecter ms_handle_connect 0x7f2f401470c0 2024-07-26T23:35:26.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.294+0000 7f2f2effd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:26.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.298+0000 7fa521c7b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-26T23:35:26.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.298+0000 7fa521c7b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-26T23:35:26.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.298+0000 7fa521c7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.298+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa51402d8f0 con 0x7fa514030880 2024-07-26T23:35:26.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.303+0000 7f3e83fff640 1 -- 172.21.15.139:0/2846045856 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f3e90062460 con 0x7f3e9c13ce20 2024-07-26T23:35:26.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.303+0000 7f3e83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.303+0000 7f3e83fff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:26.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.305+0000 7f16017fa640 1 -- 172.21.15.139:0/1013571710 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f15f8083490 con 0x7f16040b48d0 2024-07-26T23:35:26.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.305+0000 7f16017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-26T23:35:26.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.305+0000 7f16017fa640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:26.305 INFO:teuthology.orchestra.run.smithi139.stdout:85899345929 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 10 librados: watch_flush enter 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 10 librados: watch_flush exit 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 20 client.4345.objecter shutdown clearing up homeless session... 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 10 client.4345.objecter successfully canceled tick 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f15e80789d0 msgr2=0x7f15e807ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f15e80789d0 0x7f15e807ae90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16041d68a0 tx=0x7f15f0009040 comp rx=0 tx=0).stop 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 10 monclient: shutdown 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16040b48d0 msgr2=0x7f160413e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16040b48d0 0x7f160413e1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1604147940 tx=0x7f15f8004730 comp rx=0 tx=0).stop 2024-07-26T23:35:26.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.308+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 shutdown_connections 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f15e80789d0 0x7f15e807ae90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f160413d5b0 0x7f1604137d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1604140e90 0x7f1604137850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 --2- 172.21.15.139:0/1013571710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16040b48d0 0x7f160413e1a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 >> 172.21.15.139:0/1013571710 conn(0x7f16040b62c0 msgr2=0x7f16040b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 shutdown_connections 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 -- 172.21.15.139:0/1013571710 wait complete. 2024-07-26T23:35:26.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.309+0000 7f160b154640 1 librados: shutdown 2024-07-26T23:35:26.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.321+0000 7f2a1b7fe640 1 -- 172.21.15.139:0/3886977699 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f2a24096ac0 con 0x7f2a3406e720 2024-07-26T23:35:26.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.322+0000 7f2a1b7fe640 10 client.4356.objecter ms_dispatch 0x7f2a34000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:26.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.322+0000 7f2a1b7fe640 3 client.4356.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:26.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.322+0000 7f2a1b7fe640 3 client.4356.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:26.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.322+0000 7f2a1b7fe640 20 client.4356.objecter dump_active .. 0 homeless 2024-07-26T23:35:26.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345931 got 85899345929 for osd.3 2024-07-26T23:35:26.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.374+0000 7f0547522640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-26T23:35:26.551 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.374+0000 7f0547522640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-26T23:35:26.551 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.374+0000 7f0547522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.551 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.374+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0520005640 con 0x7f0540147410 2024-07-26T23:35:26.552 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.501+0000 7f3ea08b3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-26T23:35:26.552 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.501+0000 7f3ea08b3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-26T23:35:26.552 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.501+0000 7f3ea08b3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:26.552 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.501+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3e9c0ad780 con 0x7f3e9c13ce20 2024-07-26T23:35:26.552 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.504+0000 7f3e83fff640 1 -- 172.21.15.139:0/2846045856 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3e90061d60 con 0x7f3e9c13ce20 2024-07-26T23:35:26.552 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.504+0000 7f3e83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-26T23:35:26.553 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.504+0000 7f3e83fff640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:26.553 INFO:teuthology.orchestra.run.smithi139.stdout:55834574854 2024-07-26T23:35:26.554 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 10 librados: watch_flush enter 2024-07-26T23:35:26.554 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 10 librados: watch_flush exit 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 20 client.4348.objecter shutdown clearing up homeless session... 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 10 client.4348.objecter successfully canceled tick 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f3e880789d0 msgr2=0x7f3e8807ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f3e880789d0 0x7f3e8807ae90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c036000 tx=0x7f3e8c008030 comp rx=0 tx=0).stop 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 10 monclient: shutdown 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e9c13ce20 msgr2=0x7f3e9c139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e9c13ce20 0x7f3e9c139200 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000d3e0 tx=0x7f3e9000d8a0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 shutdown_connections 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f3e880789d0 0x7f3e8807ae90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.555 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f3e9c13ca20 0x7f3e9c146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e9c13ce20 0x7f3e9c139200 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 --2- 172.21.15.139:0/2846045856 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e9c0b1300 0x7f3e9c1403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 >> 172.21.15.139:0/2846045856 conn(0x7f3e9c0b1c10 msgr2=0x7f3e9c0b3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 shutdown_connections 2024-07-26T23:35:26.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 -- 172.21.15.139:0/2846045856 wait complete. 2024-07-26T23:35:26.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.507+0000 7f3ea08b3640 1 librados: shutdown 2024-07-26T23:35:26.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.5 2024-07-26T23:35:26.571 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.570+0000 7f0ee1ffb640 1 -- 172.21.15.139:0/724264485 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f0ee8011d10 con 0x7f0eec0b41c0 2024-07-26T23:35:26.571 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.570+0000 7f0ee1ffb640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0ed00789f0 0x7f0ed007aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.571 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.571+0000 7f0ef0b2c640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0ed00789f0 0x7f0ed007aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.572 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.571+0000 7f0ef0b2c640 10 monclient: get_auth_request con 0x7f0ed00789f0 auth_method 0 2024-07-26T23:35:26.572 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.571+0000 7f0ef0b2c640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0ed00789f0 0x7f0ed007aeb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0edc036000 tx=0x7f0edc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.572 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.571+0000 7f0ee1ffb640 10 client.4307.objecter ms_handle_connect 0x7f0ed00789f0 2024-07-26T23:35:26.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 1 -- 172.21.15.139:0/724264485 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f0ee8019dd0 con 0x7f0eec0b41c0 2024-07-26T23:35:26.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 10 client.4307.objecter ms_dispatch 0x7f0eec000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:26.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 3 client.4307.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:26.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 3 client.4307.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:26.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 20 client.4307.objecter dump_active .. 0 homeless 2024-07-26T23:35:26.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 1 -- 172.21.15.139:0/724264485 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f0eec14ab30 con 0x7f0eec0b41c0 2024-07-26T23:35:26.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.586+0000 7f0ee1ffb640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:26.593 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 1 -- 172.21.15.139:0/2201736133 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34032860 con 0x7f2f401470c0 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 10 monclient: dump: 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.594 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 10 monclient: _finish_auth 0 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 1 -- 172.21.15.139:0/2201736133 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f34032a00 con 0x7f2f401470c0 2024-07-26T23:35:26.595 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f448cd640 5 monclient: authenticate success, global_id 4310 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f2effd640 1 -- 172.21.15.139:0/2201736133 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34032df0 con 0x7f2f401470c0 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f448cd640 10 monclient: _renew_subs 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f448cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f40143fe0 con 0x7f2f401470c0 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.592+0000 7f2f3cd99640 4 set_mon_vals no callback set 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f2effd640 10 monclient: dump: 2024-07-26T23:35:26.596 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f448cd640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f448cd640 10 monclient: _renew_subs 2024-07-26T23:35:26.597 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f448cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4013a3e0 con 0x7f2f401470c0 2024-07-26T23:35:26.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f2f448cd640 1 librados: init done 2024-07-26T23:35:26.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f0535ffb640 1 -- 172.21.15.139:0/1584814733 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0538066030 con 0x7f0540147410 2024-07-26T23:35:26.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f0535ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-26T23:35:26.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.593+0000 7f0535ffb640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:26.598 INFO:teuthology.orchestra.run.smithi139.stdout:90194313224 2024-07-26T23:35:26.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.594+0000 7f2f448cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.594+0000 7f2f448cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.594+0000 7f2f448cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.594+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f400b45e0 con 0x7f2f401470c0 2024-07-26T23:35:26.600 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 10 librados: watch_flush enter 2024-07-26T23:35:26.600 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 10 librados: watch_flush exit 2024-07-26T23:35:26.600 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 20 client.4304.objecter shutdown clearing up homeless session... 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 10 client.4304.objecter successfully canceled tick 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f05240789a0 msgr2=0x7f052407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f05240789a0 0x7f052407ae60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f053c009990 tx=0x7f053c009110 comp rx=0 tx=0).stop 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 10 monclient: shutdown 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540147410 msgr2=0x7f05401dd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540147410 0x7f05401dd1b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f053800b9a0 tx=0x7f053800bda0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 shutdown_connections 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f05240789a0 0x7f052407ae60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0540147410 0x7f05401dd1b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540140e80 0x7f0540072030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.602 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 --2- 172.21.15.139:0/1584814733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0540068740 0x7f0540073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.602 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 >> 172.21.15.139:0/1584814733 conn(0x7f054006c720 msgr2=0x7f0540141bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.602 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 shutdown_connections 2024-07-26T23:35:26.602 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 -- 172.21.15.139:0/1584814733 wait complete. 2024-07-26T23:35:26.602 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.596+0000 7f0547522640 1 librados: shutdown 2024-07-26T23:35:26.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313226 got 90194313224 for osd.2 2024-07-26T23:35:26.637 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.636+0000 7f2f2effd640 1 -- 172.21.15.139:0/2201736133 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f2f3404c060 con 0x7f2f401470c0 2024-07-26T23:35:26.637 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.637+0000 7f2f2effd640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2f20078a20 0x7f2f2007aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.638 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.637+0000 7f2a1b7fe640 1 -- 172.21.15.139:0/3886977699 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f2a24061820 con 0x7f2a3406e720 2024-07-26T23:35:26.638 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.637+0000 7f2a1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.638 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.637+0000 7f2a1b7fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:26.639 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.637+0000 7f2f3e59c640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2f20078a20 0x7f2f2007aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.639 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.637+0000 7f2f3e59c640 10 monclient: get_auth_request con 0x7f2f20078a20 auth_method 0 2024-07-26T23:35:26.639 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.638+0000 7f2f2effd640 10 client.4310.objecter ms_handle_connect 0x7f2f20078a20 2024-07-26T23:35:26.639 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.638+0000 7f2f3e59c640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2f20078a20 0x7f2f2007aee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2f30001560 tx=0x7f2f30007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.659 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.659+0000 7f6f14ff9640 1 -- 172.21.15.139:0/4171605067 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f6f0c01a070 con 0x7f6f18140e90 2024-07-26T23:35:26.660 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.660+0000 7f6f14ff9640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6f040789a0 0x7f6f0407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:26.660 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.660+0000 7f6f17fff640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6f040789a0 0x7f6f0407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:26.661 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.660+0000 7f6f17fff640 10 monclient: get_auth_request con 0x7f6f040789a0 auth_method 0 2024-07-26T23:35:26.661 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.661+0000 7f6f14ff9640 10 client.4359.objecter ms_handle_connect 0x7f6f040789a0 2024-07-26T23:35:26.661 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.661+0000 7f6f17fff640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6f040789a0 0x7f6f0407ae60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009710 tx=0x7f6f08006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:26.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 1 -- 172.21.15.139:0/4171605067 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f6f0c05ca60 con 0x7f6f18140e90 2024-07-26T23:35:26.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 10 client.4359.objecter ms_dispatch 0x7f6f18000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:26.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 3 client.4359.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:26.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 3 client.4359.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:26.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 20 client.4359.objecter dump_active .. 0 homeless 2024-07-26T23:35:26.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 1 -- 172.21.15.139:0/4171605067 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f6f0c021080 con 0x7f6f18140e90 2024-07-26T23:35:26.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.678 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.676+0000 7f6f14ff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:26.709 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.709+0000 7fa5197fa640 1 -- 172.21.15.139:0/2746668543 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa510061260 con 0x7fa514030880 2024-07-26T23:35:26.709 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.709+0000 7fa5197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-26T23:35:26.710 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.709+0000 7fa5197fa640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:26.710 INFO:teuthology.orchestra.run.smithi139.stdout:64424509447 2024-07-26T23:35:26.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 10 librados: watch_flush enter 2024-07-26T23:35:26.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 10 librados: watch_flush exit 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 20 client.4350.objecter shutdown clearing up homeless session... 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 10 client.4350.objecter successfully canceled tick 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa500078a20 msgr2=0x7fa50007aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa500078a20 0x7fa50007aee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa508036000 tx=0x7fa508030000 comp rx=0 tx=0).stop 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 10 monclient: shutdown 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514030880 msgr2=0x7fa5140c0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514030880 0x7fa5140c0850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa51000c410 tx=0x7fa51000c810 comp rx=0 tx=0).stop 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 shutdown_connections 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa500078a20 0x7fa50007aee0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa5140c45e0 0x7fa5140c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa514031a20 0x7fa5140c0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.713 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 --2- 172.21.15.139:0/2746668543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa514030880 0x7fa5140c0850 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 >> 172.21.15.139:0/2746668543 conn(0x7fa514006010 msgr2=0x7fa514006c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.711+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 shutdown_connections 2024-07-26T23:35:26.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.712+0000 7fa521c7b640 1 -- 172.21.15.139:0/2746668543 wait complete. 2024-07-26T23:35:26.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.712+0000 7fa521c7b640 1 librados: shutdown 2024-07-26T23:35:26.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.7 2024-07-26T23:35:26.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.840+0000 7f2a394b5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-26T23:35:26.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.840+0000 7f2a394b5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-26T23:35:26.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.840+0000 7f2a394b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.840 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.840+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2a341dc990 con 0x7f2a3406e720 2024-07-26T23:35:26.843 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.843+0000 7f2a1b7fe640 1 -- 172.21.15.139:0/3886977699 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2a24061120 con 0x7f2a3406e720 2024-07-26T23:35:26.843 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.843+0000 7f2a1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-26T23:35:26.844 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.843+0000 7f2a1b7fe640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:26.844 INFO:teuthology.orchestra.run.smithi139.stdout:90194313227 2024-07-26T23:35:26.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 10 librados: watch_flush enter 2024-07-26T23:35:26.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 10 librados: watch_flush exit 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 20 client.4356.objecter shutdown clearing up homeless session... 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 10 client.4356.objecter successfully canceled tick 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2a200789d0 msgr2=0x7f2a2007ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2a200789d0 0x7f2a2007ae90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2a3413f750 tx=0x7f2a28007990 comp rx=0 tx=0).stop 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 10 monclient: shutdown 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3406e720 msgr2=0x7f2a3413f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3406e720 0x7f2a3413f9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400b4a0 tx=0x7f2a2400b960 comp rx=0 tx=0).stop 2024-07-26T23:35:26.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 shutdown_connections 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2a200789d0 0x7f2a2007ae90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2a34144cd0 0x7f2a341dcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a34143d00 0x7f2a3413d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 --2- 172.21.15.139:0/3886977699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2a3406e720 0x7f2a3413f9f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.845+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 >> 172.21.15.139:0/3886977699 conn(0x7f2a340b9c10 msgr2=0x7f2a340b6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.846+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 shutdown_connections 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.846+0000 7f2a394b5640 1 -- 172.21.15.139:0/3886977699 wait complete. 2024-07-26T23:35:26.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.846+0000 7f2a394b5640 1 librados: shutdown 2024-07-26T23:35:26.858 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313229 got 90194313227 for osd.0 2024-07-26T23:35:26.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.858+0000 7f0ef3591640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-26T23:35:26.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.858+0000 7f0ef3591640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-26T23:35:26.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.858+0000 7f0ef3591640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:26.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.858+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0eec0b45e0 con 0x7f0eec0b41c0 2024-07-26T23:35:26.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.891+0000 7f6f1e92d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-26T23:35:26.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.891+0000 7f6f1e92d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-26T23:35:26.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.891+0000 7f6f1e92d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:26.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.891+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6f18140950 con 0x7f6f18140e90 2024-07-26T23:35:26.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.901+0000 7f6f14ff9640 1 -- 172.21.15.139:0/4171605067 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6f0c069060 con 0x7f6f18140e90 2024-07-26T23:35:26.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.901+0000 7f6f14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-26T23:35:26.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.901+0000 7f6f14ff9640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:26.985 INFO:teuthology.orchestra.run.smithi139.stdout:85899345928 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 10 librados: watch_flush enter 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 10 librados: watch_flush exit 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 20 client.4359.objecter shutdown clearing up homeless session... 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 10 client.4359.objecter successfully canceled tick 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6f040789a0 msgr2=0x7f6f0407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6f040789a0 0x7f6f0407ae60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009710 tx=0x7f6f08006c90 comp rx=0 tx=0).stop 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 10 monclient: shutdown 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f18140e90 msgr2=0x7f6f18143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f18140e90 0x7f6f18143690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00c3c0 tx=0x7f6f0c00c880 comp rx=0 tx=0).stop 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 shutdown_connections 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6f040789a0 0x7f6f0407ae60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f181470c0 0x7f6f181358a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f18140e90 0x7f6f18143690 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 --2- 172.21.15.139:0/4171605067 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6f180b48d0 0x7f6f18145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 >> 172.21.15.139:0/4171605067 conn(0x7f6f180b62c0 msgr2=0x7f6f18068c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 shutdown_connections 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 -- 172.21.15.139:0/4171605067 wait complete. 2024-07-26T23:35:26.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.903+0000 7f6f1e92d640 1 librados: shutdown 2024-07-26T23:35:26.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.1 2024-07-26T23:35:26.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.949+0000 7f0ee1ffb640 1 -- 172.21.15.139:0/724264485 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ee8062340 con 0x7f0eec0b41c0 2024-07-26T23:35:26.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.949+0000 7f0ee1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-26T23:35:26.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.949+0000 7f0ee1ffb640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:26.989 INFO:teuthology.orchestra.run.smithi139.stdout:55834574854 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.951+0000 7f0ef3591640 10 librados: watch_flush enter 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.951+0000 7f0ef3591640 10 librados: watch_flush exit 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.951+0000 7f0ef3591640 20 client.4307.objecter shutdown clearing up homeless session... 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.951+0000 7f0ef3591640 10 client.4307.objecter successfully canceled tick 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0ed00789f0 msgr2=0x7f0ed007aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0ed00789f0 0x7f0ed007aeb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0edc036000 tx=0x7f0edc030000 comp rx=0 tx=0).stop 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 10 monclient: shutdown 2024-07-26T23:35:26.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:26.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0eec0b41c0 msgr2=0x7f0eec1dc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:26.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0eec0b41c0 0x7f0eec1dc960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800c430 tx=0x7f0ee800c8f0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 shutdown_connections 2024-07-26T23:35:26.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0ed00789f0 0x7f0ed007aeb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0eec14c1d0 0x7f0eec138c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eec1470c0 0x7f0eec145180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 --2- 172.21.15.139:0/724264485 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0eec0b41c0 0x7f0eec1dc960 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:26.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 >> 172.21.15.139:0/724264485 conn(0x7f0eec0b62c0 msgr2=0x7f0eec13eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:26.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 shutdown_connections 2024-07-26T23:35:26.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 -- 172.21.15.139:0/724264485 wait complete. 2024-07-26T23:35:26.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.952+0000 7f0ef3591640 1 librados: shutdown 2024-07-26T23:35:26.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.6 2024-07-26T23:35:26.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 1 -- 172.21.15.139:0/2201736133 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f2f3407ef50 con 0x7f2f401470c0 2024-07-26T23:35:26.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 10 client.4310.objecter ms_dispatch 0x7f2f40000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:26.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 3 client.4310.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:26.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 3 client.4310.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:26.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 20 client.4310.objecter dump_active .. 0 homeless 2024-07-26T23:35:26.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 1 -- 172.21.15.139:0/2201736133 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f2f34040070 con 0x7f2f401470c0 2024-07-26T23:35:26.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:26.996 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:26.962+0000 7f2f2effd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:27.190 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.189+0000 7f2f448cd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-26T23:35:27.190 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.189+0000 7f2f448cd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-26T23:35:27.190 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.189+0000 7f2f448cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.191 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.189+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2f401403e0 con 0x7f2f401470c0 2024-07-26T23:35:27.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.201+0000 7f2f2effd640 1 -- 172.21.15.139:0/2201736133 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2f340833c0 con 0x7f2f401470c0 2024-07-26T23:35:27.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.201+0000 7f2f2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-07-26T23:35:27.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.201+0000 7f2f2effd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:27.203 INFO:teuthology.orchestra.run.smithi139.stdout:77309411338 2024-07-26T23:35:27.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 10 librados: watch_flush enter 2024-07-26T23:35:27.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 10 librados: watch_flush exit 2024-07-26T23:35:27.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 20 client.4310.objecter shutdown clearing up homeless session... 2024-07-26T23:35:27.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 10 client.4310.objecter successfully canceled tick 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2f20078a20 msgr2=0x7f2f2007aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2f20078a20 0x7f2f2007aee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2f30001560 tx=0x7f2f30007990 comp rx=0 tx=0).stop 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 10 monclient: shutdown 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2f401470c0 msgr2=0x7f2f401dc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2f401470c0 0x7f2f401dc450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2f340090b0 tx=0x7f2f340046c0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 shutdown_connections 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f2f20078a20 0x7f2f2007aee0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2f401470c0 0x7f2f401dc450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f4014c1d0 0x7f2f40139ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 --2- 172.21.15.139:0/2201736133 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2f400b41c0 0x7f2f401dd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 >> 172.21.15.139:0/2201736133 conn(0x7f2f400b62c0 msgr2=0x7f2f400b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:27.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 shutdown_connections 2024-07-26T23:35:27.208 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.204+0000 7f2f448cd640 1 -- 172.21.15.139:0/2201736133 wait complete. 2024-07-26T23:35:27.208 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.205+0000 7f2f448cd640 1 librados: shutdown 2024-07-26T23:35:27.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411338 for osd.4 2024-07-26T23:35:27.219 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:27.326 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.425+0000 7f794e2f2640 10 monclient: get_monmap_and_config 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.425+0000 7f794e2f2640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.425+0000 7f794e2f2640 10 monclient: monmap: 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:27.426817+0000 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:27.426817+0000 2024-07-26T23:35:27.427 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.428 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.428 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:27.428 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:27.428 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:27.428 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:27.428 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:27.428 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.426+0000 7f794e2f2640 1 Processor -- start 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.426+0000 7f794e2f2640 1 -- start start 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.426+0000 7f794e2f2640 10 monclient: init 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f7948068740 0x7f794813d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): picked mon.noname-a con 0x7f7948068740 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): picked mon.noname-b con 0x7f794806fa60 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7948143910 0x7f7948143d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): picked mon.noname-c con 0x7f7948143910 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:27.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f7948068b20 con 0x7f7948068740 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f79480af8f0 con 0x7f7948143910 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f79480b0230 con 0x7f794806fa60 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794e2f2640 10 monclient(hunting): authenticate will time out at 1360.463197s 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f79477fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f7947fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f7948068740 0x7f794813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.427+0000 7f794c88f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7948143910 0x7f7948143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60848/0 (socket says 172.21.15.139:60848) 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 1 -- 172.21.15.139:0/1013318603 learned_addr learned my addr 172.21.15.139:0/1013318603 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:27.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 10 monclient(hunting): get_auth_request con 0x7f794806fa60 auth_method 0 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f7947fff640 10 monclient(hunting): get_auth_request con 0x7f7948068740 auth_method 0 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f7947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f7947fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f7947fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f794c88f640 10 monclient(hunting): get_auth_request con 0x7f7948143910 auth_method 0 2024-07-26T23:35:27.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f794c88f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f794c88f640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f794c88f640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f79477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f794c88f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f794c88f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f794c88f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f7947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f7947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.428+0000 7f7947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-07-26T23:35:27.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 1 -- 172.21.15.139:0/1013318603 >> v2:172.21.15.177:3300/0 conn(0x7f7948143910 msgr2=0x7f7948143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:27.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 1 --2- 172.21.15.139:0/1013318603 >> v2:172.21.15.177:3300/0 conn(0x7f7948143910 0x7f7948143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 1 -- 172.21.15.139:0/1013318603 >> v2:172.21.15.139:3300/0 conn(0x7f7948068740 msgr2=0x7f794813d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:27.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 1 --2- 172.21.15.139:0/1013318603 >> v2:172.21.15.139:3300/0 conn(0x7f7948068740 0x7f794813d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 1 monclient: found mon.noname-b 2024-07-26T23:35:27.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.430+0000 7f79477fe640 1 -- 172.21.15.139:0/1013318603 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794806d950 con 0x7f794806fa60 2024-07-26T23:35:27.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.432+0000 7f79477fe640 1 --2- 172.21.15.139:0/1013318603 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7948069870 tx=0x7f793c02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b323f185acfde608 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 1 -- 172.21.15.139:0/1013318603 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c02e070 con 0x7f794806fa60 2024-07-26T23:35:27.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:27.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: dump: 2024-07-26T23:35:27.437 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.437 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 1 -- 172.21.15.139:0/1013318603 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f793c037070 con 0x7f794806fa60 2024-07-26T23:35:27.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 5 monclient: authenticate success, global_id 4313 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 1 -- 172.21.15.139:0/1013318603 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c031a00 con 0x7f794806fa60 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 4 set_mon_vals no callback set 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f7946ffd640 10 monclient: dump: 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.439 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.440 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.440 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.440 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.440 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.440 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 10 monclient: shutdown 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 -- 172.21.15.139:0/1013318603 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 msgr2=0x7f794813d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1013318603 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7948069870 tx=0x7f793c02b880 comp rx=0 tx=0).stop 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 -- 172.21.15.139:0/1013318603 shutdown_connections 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1013318603 >> v2:172.21.15.177:3300/0 conn(0x7f7948143910 0x7f7948143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1013318603 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1013318603 >> v2:172.21.15.139:3300/0 conn(0x7f7948068740 0x7f794813d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 -- 172.21.15.139:0/1013318603 >> 172.21.15.139:0/1013318603 conn(0x7f794806c720 msgr2=0x7f794806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 -- 172.21.15.139:0/1013318603 shutdown_connections 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.436+0000 7f794e2f2640 1 -- 172.21.15.139:0/1013318603 wait complete. 2024-07-26T23:35:27.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.437+0000 7f794e2f2640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.437+0000 7f794e2f2640 10 monclient: monmap: 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:27.445 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.437+0000 7f794e2f2640 1 librados: starting msgr at 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.437+0000 7f794e2f2640 1 librados: starting objecter 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 Processor -- start 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 -- start start 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 librados: setting wanted keys 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 librados: calling monclient init 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient: init 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7948068740 0x7f794813ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): picked mon.noname-b con 0x7f7948068740 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:27.446 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): picked mon.noname-c con 0x7f794806fa60 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7948143910 0x7f79480b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): picked mon.noname-a con 0x7f7948143910 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7948144440 con 0x7f7948143910 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f79481442c0 con 0x7f7948068740 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f7948146e90 con 0x7f794806fa60 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.438+0000 7f794e2f2640 10 monclient(hunting): authenticate will time out at 1360.474897s 2024-07-26T23:35:27.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f79477fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f794c88f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7948143910 0x7f79480b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f79477fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60852/0 (socket says 172.21.15.139:60852) 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f7947fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7948068740 0x7f794813ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f79477fe640 1 -- 172.21.15.139:0/1611537543 learned_addr learned my addr 172.21.15.139:0/1611537543 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f79477fe640 10 monclient(hunting): get_auth_request con 0x7f794806fa60 auth_method 0 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f79477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f79477fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f79477fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f794c88f640 10 monclient(hunting): get_auth_request con 0x7f7948143910 auth_method 0 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f794c88f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f794c88f640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f794c88f640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f7947fff640 10 monclient(hunting): get_auth_request con 0x7f7948068740 auth_method 0 2024-07-26T23:35:27.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f7947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f7947fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.439+0000 7f7947fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f79477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f79477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f79477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f794c88f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f794c88f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f794c88f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f7947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f7947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.440+0000 7f7947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-07-26T23:35:27.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 1 -- 172.21.15.139:0/1611537543 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 msgr2=0x7f794813e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 1 -- 172.21.15.139:0/1611537543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7948143910 msgr2=0x7f79480b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7948143910 0x7f79480b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 1 monclient: found mon.noname-b 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f7947fff640 1 -- 172.21.15.139:0/1611537543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7948140450 con 0x7f7948068740 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.443+0000 7f79477fe640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.445+0000 7f7947fff640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7948068740 0x7f794813ee60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f793800e8a0 tx=0x7f793800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.445+0000 7f7944ff9640 10 client.?.objecter ms_handle_connect 0x7f7948068740 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.445+0000 7f7944ff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:27.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 1 -- 172.21.15.139:0/1611537543 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938017070 con 0x7f7948068740 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: dump: 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.453 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 1 -- 172.21.15.139:0/1611537543 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f793800c940 con 0x7f7948068740 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 1 -- 172.21.15.139:0/1611537543 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793800ccf0 con 0x7f7948068740 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 5 monclient: authenticate success, global_id 4372 2024-07-26T23:35:27.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 10 monclient: _renew_subs 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79481d6790 con 0x7f7948068740 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f7944ff9640 10 monclient: dump: 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.455 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f79467fc640 4 set_mon_vals no callback set 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 10 monclient: _renew_subs 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79480b4770 con 0x7f7948068740 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.449+0000 7f794e2f2640 1 librados: init done 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.450+0000 7f794e2f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.450+0000 7f794e2f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.450+0000 7f794e2f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.456 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.450+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7948143d80 con 0x7f7948068740 2024-07-26T23:35:27.558 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-07-26T23:35:27.609 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-07-26T23:35:27.666 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.664+0000 7fd8d50e4640 10 monclient: get_monmap_and_config 2024-07-26T23:35:27.666 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.664+0000 7fd8d50e4640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.666 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.664+0000 7fd8d50e4640 10 monclient: monmap: 2024-07-26T23:35:27.666 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.666 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.666 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:27.665522+0000 2024-07-26T23:35:27.666 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:27.665522+0000 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 1 Processor -- start 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 1 -- start start 2024-07-26T23:35:27.667 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient: init 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8d01470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8d00b41c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fd8d014d1e0 0x7fd8d014f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd8d014d1e0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8d0147ce0 con 0x7fd8d01470c0 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8d00acaf0 con 0x7fd8d00b41c0 2024-07-26T23:35:27.669 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fd8d01477c0 con 0x7fd8d014d1e0 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.665+0000 7fd8d50e4640 10 monclient(hunting): authenticate will time out at 1360.701818s 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.666+0000 7fd8cf59e640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fd8d014d1e0 0x7fd8d014f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.667+0000 7fd8cf59e640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fd8d014d1e0 0x7fd8d014f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:32846/0 (socket says 172.21.15.139:32846) 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.667+0000 7fd8cf59e640 1 -- 172.21.15.139:0/1835903862 learned_addr learned my addr 172.21.15.139:0/1835903862 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.667+0000 7fd8cf59e640 10 monclient(hunting): get_auth_request con 0x7fd8d014d1e0 auth_method 0 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.667+0000 7fd8cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.667+0000 7fd8cf59e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.667+0000 7fd8cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.668+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.668+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.668+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-07-26T23:35:27.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 1 -- 172.21.15.139:0/1835903862 >> v1:172.21.15.177:6789/0 conn(0x7fd8d00b41c0 legacy=0x7fd8d00b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:27.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 1 -- 172.21.15.139:0/1835903862 >> v1:172.21.15.139:6789/0 conn(0x7fd8d01470c0 legacy=0x7fd8d0140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 1 monclient: found mon.noname-a 2024-07-26T23:35:27.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:27.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.671+0000 7fd8cf59e640 1 -- 172.21.15.139:0/1835903862 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d01474a0 con 0x7fd8d014d1e0 2024-07-26T23:35:27.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.674+0000 7fd8cf59e640 1 --2- 172.21.15.139:0/1835903862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d014f5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009e50 tx=0x7fd8c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a962ccfe85783801 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.679 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 1 -- 172.21.15.139:0/1835903862 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c802ca70 con 0x7fd8d014d1e0 2024-07-26T23:35:27.679 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.679 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:27.679 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: dump: 2024-07-26T23:35:27.679 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.679 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.679 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 1 -- 172.21.15.139:0/1835903862 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c802cc10 con 0x7fd8d014d1e0 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8d50e4640 5 monclient: authenticate success, global_id 4368 2024-07-26T23:35:27.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8d50e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8d50e4640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 1 -- 172.21.15.139:0/1835903862 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c80343f0 con 0x7fd8d014d1e0 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8d50e4640 4 set_mon_vals no callback set 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.678+0000 7fd8cdd9b640 10 monclient: dump: 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.681 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.682 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.682 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.682 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.682 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 10 monclient: shutdown 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 1 -- 172.21.15.139:0/1835903862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 msgr2=0x7fd8d014f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/1835903862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d014f5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009e50 tx=0x7fd8c802bed0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 1 -- 172.21.15.139:0/1835903862 shutdown_connections 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/1835903862 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d014f5d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 1 -- 172.21.15.139:0/1835903862 >> 172.21.15.139:0/1835903862 conn(0x7fd8d00b62c0 msgr2=0x7fd8d00b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:27.685 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 1 -- 172.21.15.139:0/1835903862 shutdown_connections 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.679+0000 7fd8d50e4640 1 -- 172.21.15.139:0/1835903862 wait complete. 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.680+0000 7fd8d50e4640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.680+0000 7fd8d50e4640 10 monclient: monmap: 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.686 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.680+0000 7fd8d50e4640 1 librados: starting msgr at 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.680+0000 7fd8d50e4640 1 librados: starting objecter 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.681+0000 7fd8d50e4640 1 Processor -- start 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.681+0000 7fd8d50e4640 1 -- start start 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.681+0000 7fd8d50e4640 1 librados: setting wanted keys 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.681+0000 7fd8d50e4640 1 librados: calling monclient init 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.681+0000 7fd8d50e4640 10 monclient: init 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.681+0000 7fd8d50e4640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.681+0000 7fd8d50e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:35:27.687 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8d00b41c0 0x7fd8d00add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd8d00b41c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd8d01470c0 0x7fd8d013f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd8d01470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd8d014d1e0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d0141410 con 0x7fd8d014d1e0 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d0074850 con 0x7fd8d00b41c0 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d0141060 con 0x7fd8d01470c0 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8d50e4640 10 monclient(hunting): authenticate will time out at 1360.718086s 2024-07-26T23:35:27.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8ce59c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd8d01470c0 0x7fd8d013f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.682+0000 7fd8cf59e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd8d01470c0 0x7fd8d013f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60854/0 (socket says 172.21.15.139:60854) 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 1 -- 172.21.15.139:0/832341492 learned_addr learned my addr 172.21.15.139:0/832341492 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 10 monclient(hunting): get_auth_request con 0x7fd8d01470c0 auth_method 0 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8cf59e640 10 monclient(hunting): get_auth_request con 0x7fd8d014d1e0 auth_method 0 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8cf59e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.683+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 1 -- 172.21.15.139:0/832341492 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd8d01470c0 msgr2=0x7fd8d013f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd8d01470c0 0x7fd8d013f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 1 -- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8d00b41c0 msgr2=0x7fd8d00add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8d00b41c0 0x7fd8d00add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 1 monclient: found mon.noname-a 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.686+0000 7fd8cf59e640 1 -- 172.21.15.139:0/832341492 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d01403e0 con 0x7fd8d014d1e0 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.689+0000 7fd8cf59e640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d013fea0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8032000 tx=0x7fd8c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.689+0000 7fd8b77fe640 10 client.?.objecter ms_handle_connect 0x7fd8d014d1e0 2024-07-26T23:35:27.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.689+0000 7fd8b77fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 1 -- 172.21.15.139:0/832341492 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8043020 con 0x7fd8d014d1e0 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: dump: 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 1 -- 172.21.15.139:0/832341492 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c803e450 con 0x7fd8d014d1e0 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 1 -- 172.21.15.139:0/832341492 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8009510 con 0x7fd8d014d1e0 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8b77fe640 10 monclient: dump: 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 5 monclient: authenticate success, global_id 4374 2024-07-26T23:35:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 10 monclient: _renew_subs 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d00af6e0 con 0x7fd8d014d1e0 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8cd59a640 4 set_mon_vals no callback set 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 10 monclient: _renew_subs 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d00ad7b0 con 0x7fd8d014d1e0 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.693+0000 7fd8d50e4640 1 librados: init done 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.694+0000 7fd8d50e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.694+0000 7fd8d50e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.694+0000 7fd8d50e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.694+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d00b45e0 con 0x7fd8d014d1e0 2024-07-26T23:35:27.704 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.696+0000 7f7944ff9640 1 -- 172.21.15.139:0/1611537543 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f793801a070 con 0x7f7948068740 2024-07-26T23:35:27.704 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.697+0000 7f7944ff9640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f79340789f0 0x7f793407aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.704 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.697+0000 7f79477fe640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f79340789f0 0x7f793407aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.704 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.697+0000 7f79477fe640 10 monclient: get_auth_request con 0x7f79340789f0 auth_method 0 2024-07-26T23:35:27.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.697+0000 7f7944ff9640 10 client.4372.objecter ms_handle_connect 0x7f79340789f0 2024-07-26T23:35:27.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.697+0000 7f79477fe640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f79340789f0 0x7f793407aeb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f793c02b8c0 tx=0x7f793c03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.700+0000 7f7944ff9640 1 -- 172.21.15.139:0/1611537543 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f793805dc80 con 0x7f7948068740 2024-07-26T23:35:27.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.700+0000 7f7944ff9640 10 client.4372.objecter ms_dispatch 0x7f7948000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:27.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.700+0000 7f7944ff9640 3 client.4372.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:27.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.700+0000 7f7944ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:27.705 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.700+0000 7f7944ff9640 20 client.4372.objecter dump_active .. 0 homeless 2024-07-26T23:35:27.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.711+0000 7f7944ff9640 1 -- 172.21.15.139:0/1611537543 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f79380619f0 con 0x7f7948068740 2024-07-26T23:35:27.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.711+0000 7f7944ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.711+0000 7f7944ff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:27.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.715+0000 7f2924b10640 10 monclient: get_monmap_and_config 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.715+0000 7f2924b10640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.715+0000 7f2924b10640 10 monclient: monmap: 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:27.716787+0000 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:27.716787+0000 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:27.717 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:27.718 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:27.718 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:27.718 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:27.718 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.715+0000 7fd8b77fe640 1 -- 172.21.15.139:0/832341492 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fd8c8046020 con 0x7fd8d014d1e0 2024-07-26T23:35:27.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.716+0000 7fd8b77fe640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd8bc0789f0 0x7fd8bc07aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.716+0000 7f2924b10640 1 Processor -- start 2024-07-26T23:35:27.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.716+0000 7f2924b10640 1 -- start start 2024-07-26T23:35:27.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.716+0000 7f2924b10640 10 monclient: init 2024-07-26T23:35:27.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.716+0000 7fd8ced9d640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd8bc0789f0 0x7fd8bc07aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.716+0000 7fd8ced9d640 10 monclient: get_auth_request con 0x7fd8bc0789f0 auth_method 0 2024-07-26T23:35:27.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7fd8ced9d640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd8bc0789f0 0x7fd8bc07aeb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0000f60 tx=0x7fd8c00079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7fd8b77fe640 10 client.4374.objecter ms_handle_connect 0x7fd8bc0789f0 2024-07-26T23:35:27.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-07-26T23:35:27.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f292006e720 0x7f2920139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient(hunting): picked mon.noname-a con 0x7f292006e720 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:27.722 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2920143d00 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f2920145100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient(hunting): picked mon.noname-b con 0x7f2920144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29200ac810 con 0x7f2920143d00 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f292006eb00 con 0x7f292006e720 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f2920144710 con 0x7f2920144cd0 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.717+0000 7f2924b10640 10 monclient(hunting): authenticate will time out at 1360.753487s 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.718+0000 7f291dd9b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f2920143d00 0x7f292013a110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:52074/0 (socket says 172.21.15.139:52074) 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.718+0000 7f291dd9b640 1 -- 172.21.15.139:0/3127738917 learned_addr learned my addr 172.21.15.139:0/3127738917 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.718+0000 7f291ed9d640 1 --2- 172.21.15.139:0/3127738917 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f2920145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.718+0000 7f291ed9d640 10 monclient(hunting): get_auth_request con 0x7f2920144cd0 auth_method 0 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.718+0000 7f291ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.718+0000 7f291ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.718+0000 7f291ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.719+0000 7f291ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.719+0000 7f291ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.719+0000 7f291ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291d59a640 1 -- 172.21.15.139:0/3127738917 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096703086 0 0) 0x7f29200ac810 con 0x7f2920143d00 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291d59a640 10 monclient(hunting): _negotiate 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291d59a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291d59a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291d59a640 10 monclient(hunting): my global_id is 4375 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291d59a640 1 -- 172.21.15.139:0/3127738917 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2908003520 con 0x7f2920143d00 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 1 -- 172.21.15.139:0/3127738917 >> v2:172.21.15.139:3300/0 conn(0x7f292006e720 msgr2=0x7f2920139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 1 --2- 172.21.15.139:0/3127738917 >> v2:172.21.15.139:3300/0 conn(0x7f292006e720 0x7f2920139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 1 -- 172.21.15.139:0/3127738917 >> v1:172.21.15.177:6789/0 conn(0x7f2920143d00 legacy=0x7f292013a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 1 monclient: found mon.noname-b 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.720+0000 7f291ed9d640 1 -- 172.21.15.139:0/3127738917 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200acaf0 con 0x7f2920144cd0 2024-07-26T23:35:27.726 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-07-26T23:35:27.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.721+0000 7fd8b77fe640 1 -- 172.21.15.139:0/832341492 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fd8c8080140 con 0x7fd8d014d1e0 2024-07-26T23:35:27.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.721+0000 7fd8b77fe640 10 client.4374.objecter ms_dispatch 0x7fd8d0000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:27.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.721+0000 7fd8b77fe640 3 client.4374.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:27.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.721+0000 7fd8b77fe640 3 client.4374.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:27.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.721+0000 7fd8b77fe640 20 client.4374.objecter dump_active .. 0 homeless 2024-07-26T23:35:27.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.722+0000 7f291ed9d640 1 --2- 172.21.15.139:0/3127738917 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f2920145100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2918009d90 tx=0x7f291802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd5252e49a3a98b5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 1 -- 172.21.15.139:0/3127738917 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918039070 con 0x7f2920144cd0 2024-07-26T23:35:27.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:27.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: dump: 2024-07-26T23:35:27.728 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.729 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 1 -- 172.21.15.139:0/3127738917 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f291802cce0 con 0x7f2920144cd0 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f2924b10640 5 monclient: authenticate success, global_id 4322 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f2924b10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f2924b10640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f2924b10640 4 set_mon_vals no callback set 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 1 -- 172.21.15.139:0/3127738917 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918034660 con 0x7f2920144cd0 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.726+0000 7f291d59a640 10 monclient: dump: 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.731 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 10 monclient: shutdown 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 -- 172.21.15.139:0/3127738917 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 msgr2=0x7f2920145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 --2- 172.21.15.139:0/3127738917 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f2920145100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2918009d90 tx=0x7f291802bf30 comp rx=0 tx=0).stop 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 -- 172.21.15.139:0/3127738917 shutdown_connections 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 --2- 172.21.15.139:0/3127738917 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f2920145100 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 --2- 172.21.15.139:0/3127738917 >> v2:172.21.15.139:3300/0 conn(0x7f292006e720 0x7f2920139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 -- 172.21.15.139:0/3127738917 >> 172.21.15.139:0/3127738917 conn(0x7f29200b9c10 msgr2=0x7f29200ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 -- 172.21.15.139:0/3127738917 shutdown_connections 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.727+0000 7f2924b10640 1 -- 172.21.15.139:0/3127738917 wait complete. 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.728+0000 7f2924b10640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.728+0000 7f2924b10640 10 monclient: monmap: 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:27.737 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 1 librados: starting msgr at 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 1 librados: starting objecter 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 1 Processor -- start 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 1 -- start start 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 1 librados: setting wanted keys 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 1 librados: calling monclient init 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 10 monclient: init 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.729+0000 7f2924b10640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:27.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f292006e720 0x7f29200727d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): picked mon.noname-b con 0x7f292006e720 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2920143d00 0x7f29201e0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): picked mon.noname-a con 0x7f2920143d00 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f29201e1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): picked mon.noname-c con 0x7f2920144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2920145640 con 0x7f2920143d00 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2920141390 con 0x7f292006e720 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f29201457c0 con 0x7f2920144cd0 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f2924b10640 10 monclient(hunting): authenticate will time out at 1360.766090s 2024-07-26T23:35:27.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f291e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f292006e720 0x7f29200727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f291ed9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f29201e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.730+0000 7f291e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f292006e720 0x7f29200727d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41470/0 (socket says 172.21.15.139:41470) 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 1 -- 172.21.15.139:0/1691464336 learned_addr learned my addr 172.21.15.139:0/1691464336 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 10 monclient(hunting): get_auth_request con 0x7f292006e720 auth_method 0 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291ed9d640 10 monclient(hunting): get_auth_request con 0x7f2920144cd0 auth_method 0 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.731+0000 7f291ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 1 -- 172.21.15.139:0/1691464336 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 msgr2=0x7f29201e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f29201e1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 1 -- 172.21.15.139:0/1691464336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2920143d00 msgr2=0x7f29201e0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2920143d00 0x7f29201e0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 1 monclient: found mon.noname-b 2024-07-26T23:35:27.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.734+0000 7f291e59c640 1 -- 172.21.15.139:0/1691464336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29201e07d0 con 0x7f292006e720 2024-07-26T23:35:27.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.736+0000 7f291e59c640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f292006e720 0x7f29200727d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f291000b4a0 tx=0x7f291000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.736+0000 7f290effd640 10 client.?.objecter ms_handle_connect 0x7f292006e720 2024-07-26T23:35:27.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.736+0000 7f290effd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:27.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 1 -- 172.21.15.139:0/1691464336 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910014070 con 0x7f292006e720 2024-07-26T23:35:27.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.745 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: dump: 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.746 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 1 -- 172.21.15.139:0/1691464336 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2910010610 con 0x7f292006e720 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 1 -- 172.21.15.139:0/1691464336 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910010a00 con 0x7f292006e720 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:27.747 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f290effd640 10 monclient: dump: 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.748 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 5 monclient: authenticate success, global_id 4378 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 10 monclient: _renew_subs 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292006fa40 con 0x7f292006e720 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f291cd99640 4 set_mon_vals no callback set 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 10 monclient: _renew_subs 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.740+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2920147f20 con 0x7f292006e720 2024-07-26T23:35:27.749 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.741+0000 7f2924b10640 1 librados: init done 2024-07-26T23:35:27.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.743+0000 7f2924b10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.743+0000 7f2924b10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.743+0000 7f2924b10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.750 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.743+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2920145180 con 0x7f292006e720 2024-07-26T23:35:27.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.761+0000 7f290effd640 1 -- 172.21.15.139:0/1691464336 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f2910017070 con 0x7f292006e720 2024-07-26T23:35:27.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.762+0000 7f290effd640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f29000789f0 0x7f290007aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.762+0000 7f291dd9b640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f29000789f0 0x7f290007aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.762+0000 7f291dd9b640 10 monclient: get_auth_request con 0x7f29000789f0 auth_method 0 2024-07-26T23:35:27.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.763+0000 7f290effd640 10 client.4378.objecter ms_handle_connect 0x7f29000789f0 2024-07-26T23:35:27.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.763+0000 7f291dd9b640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f29000789f0 0x7f290007aeb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2914001220 tx=0x7f29140078f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.765+0000 7f290effd640 1 -- 172.21.15.139:0/1691464336 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f291005d410 con 0x7f292006e720 2024-07-26T23:35:27.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.765+0000 7f290effd640 10 client.4378.objecter ms_dispatch 0x7f2920000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:27.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.765+0000 7f290effd640 3 client.4378.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:27.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.765+0000 7f290effd640 3 client.4378.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:27.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.766+0000 7f290effd640 20 client.4378.objecter dump_active .. 0 homeless 2024-07-26T23:35:27.835 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.834+0000 7f4259457640 10 monclient: get_monmap_and_config 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.834+0000 7f4259457640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.834+0000 7f4259457640 10 monclient: monmap: 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:27.835258+0000 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:27.835258+0000 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:27.836 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:27.837 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.834+0000 7f4259457640 1 Processor -- start 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 1 -- start start 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient: init 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42541470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42540b41c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f425414d1e0 0x7f425414f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient(hunting): picked mon.noname-c con 0x7f425414d1e0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:27.838 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4254147ce0 con 0x7f42541470c0 2024-07-26T23:35:27.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42540acaf0 con 0x7f42540b41c0 2024-07-26T23:35:27.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f42541477c0 con 0x7f425414d1e0 2024-07-26T23:35:27.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.835+0000 7f4259457640 10 monclient(hunting): authenticate will time out at 1360.871754s 2024-07-26T23:35:27.858 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-07-26T23:35:27.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.933+0000 7f794e2f2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-26T23:35:27.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.933+0000 7f794e2f2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-26T23:35:27.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.933+0000 7f794e2f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:27.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.933+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7948069500 con 0x7f7948068740 2024-07-26T23:35:27.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.959+0000 7f23b9bcf640 10 monclient: get_monmap_and_config 2024-07-26T23:35:27.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.959+0000 7f23b9bcf640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.959+0000 7f23b9bcf640 10 monclient: monmap: 2024-07-26T23:35:27.960 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.960 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:27.960148+0000 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:27.960148+0000 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:27.961 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.959+0000 7f23b9bcf640 1 Processor -- start 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 1 -- start start 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient: init 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f23b41470c0 0x7f23b4140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f23b41470c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23b4140e90 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:27.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b40b48d0 0x7f23b414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f23b40b48d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23b4147ce0 con 0x7f23b4140e90 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f23b40acaf0 con 0x7f23b41470c0 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f23b41477c0 con 0x7f23b40b48d0 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.960+0000 7f23b9bcf640 10 monclient(hunting): authenticate will time out at 1360.996737s 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.961+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b40b48d0 0x7f23b414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.961+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b40b48d0 0x7f23b414e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60880/0 (socket says 172.21.15.139:60880) 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.961+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390054565 learned_addr learned my addr 172.21.15.139:0/3390054565 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.961+0000 7f23b3fff640 10 monclient(hunting): get_auth_request con 0x7f23b40b48d0 auth_method 0 2024-07-26T23:35:27.964 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.961+0000 7f23b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.965 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.961+0000 7f23b3fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.965 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.961+0000 7f23b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.965 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.962+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.965 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.962+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.965 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.962+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390054565 >> v2:172.21.15.139:3300/0 conn(0x7f23b41470c0 msgr2=0x7f23b4140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 1 --2- 172.21.15.139:0/3390054565 >> v2:172.21.15.139:3300/0 conn(0x7f23b41470c0 0x7f23b4140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390054565 >> v1:172.21.15.177:6789/0 conn(0x7f23b4140e90 legacy=0x7f23b40b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 1 monclient: found mon.noname-b 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.966 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.964+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390054565 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b41474a0 con 0x7f23b40b48d0 2024-07-26T23:35:27.967 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.965+0000 7f23b3fff640 1 --2- 172.21.15.139:0/3390054565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b40b48d0 0x7f23b414e1e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009870 tx=0x7f23a402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebc79ea44de58b64 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 1 -- 172.21.15.139:0/3390054565 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a40317c0 con 0x7f23b40b48d0 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: dump: 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.972 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 1 -- 172.21.15.139:0/3390054565 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23a4037070 con 0x7f23b40b48d0 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b9bcf640 5 monclient: authenticate success, global_id 4328 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b9bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b9bcf640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b9bcf640 4 set_mon_vals no callback set 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 1 -- 172.21.15.139:0/3390054565 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4031b80 con 0x7f23b40b48d0 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:27.973 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.969+0000 7f23b27fc640 10 monclient: dump: 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 10 monclient: shutdown 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:27.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390054565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b40b48d0 msgr2=0x7f23b414e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390054565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b40b48d0 0x7f23b414e1e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009870 tx=0x7f23a402b860 comp rx=0 tx=0).stop 2024-07-26T23:35:27.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390054565 shutdown_connections 2024-07-26T23:35:27.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390054565 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b40b48d0 0x7f23b414e1e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390054565 >> v2:172.21.15.139:3300/0 conn(0x7f23b41470c0 0x7f23b4140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390054565 >> 172.21.15.139:0/3390054565 conn(0x7f23b40b62c0 msgr2=0x7f23b40b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:27.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390054565 shutdown_connections 2024-07-26T23:35:27.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.970+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390054565 wait complete. 2024-07-26T23:35:27.977 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 10 monclient: build_initial_monmap 2024-07-26T23:35:27.977 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 10 monclient: monmap: 2024-07-26T23:35:27.977 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:27.977 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:27.977 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:27.977 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 1 librados: starting msgr at 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 1 librados: starting objecter 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 1 Processor -- start 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 1 -- start start 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 1 librados: setting wanted keys 2024-07-26T23:35:27.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 1 librados: calling monclient init 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.971+0000 7f23b9bcf640 10 monclient: init 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b40b48d0 0x7f23b413d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f23b40b48d0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23b4140e90 0x7f23b413df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f23b4140e90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b41470c0 0x7f23b4142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f23b41470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f23b4141410 con 0x7f23b40b48d0 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.979 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f23b4074850 con 0x7f23b4140e90 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f23b414e720 con 0x7f23b41470c0 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b9bcf640 10 monclient(hunting): authenticate will time out at 1361.008296s 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.972+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b41470c0 0x7f23b4142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.973+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b41470c0 0x7f23b4142ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60886/0 (socket says 172.21.15.139:60886) 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.973+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390344634 learned_addr learned my addr 172.21.15.139:0/3390344634 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.973+0000 7f23b3fff640 10 monclient(hunting): get_auth_request con 0x7f23b41470c0 auth_method 0 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.973+0000 7f23b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.973+0000 7f23b3fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.973+0000 7f23b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:27.980 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.974+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.974+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.974+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23b4140e90 msgr2=0x7f23b413df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23b4140e90 0x7f23b413df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390344634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b40b48d0 msgr2=0x7f23b413d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b40b48d0 0x7f23b413d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:27.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:27.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 1 monclient: found mon.noname-c 2024-07-26T23:35:27.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:27.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.978+0000 7f23b3fff640 1 -- 172.21.15.139:0/3390344634 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b413fa80 con 0x7f23b41470c0 2024-07-26T23:35:27.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.981+0000 7f23b3fff640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b41470c0 0x7f23b4142ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009870 tx=0x7f23a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:27.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.981+0000 7f23abfff640 10 client.?.objecter ms_handle_connect 0x7f23b41470c0 2024-07-26T23:35:27.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.981+0000 7f23abfff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:27.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 1 -- 172.21.15.139:0/3390344634 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4032860 con 0x7f23b41470c0 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: dump: 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.988 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: _finish_auth 0 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 1 -- 172.21.15.139:0/3390344634 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23a4032a00 con 0x7f23b41470c0 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 5 monclient: authenticate success, global_id 4331 2024-07-26T23:35:27.989 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 10 monclient: _renew_subs 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b413fd60 con 0x7f23b41470c0 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 1 -- 172.21.15.139:0/3390344634 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4032db0 con 0x7f23b41470c0 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23abfff640 10 monclient: dump: 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:27.990 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b1ffb640 4 set_mon_vals no callback set 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 10 monclient: _renew_subs 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b413ead0 con 0x7f23b41470c0 2024-07-26T23:35:27.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.986+0000 7f23b9bcf640 1 librados: init done 2024-07-26T23:35:27.991 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-07-26T23:35:27.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.987+0000 7f23b9bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.987+0000 7f23b9bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:27.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.987+0000 7f23b9bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:27.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:27.987+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23b4140950 con 0x7f23b41470c0 2024-07-26T23:35:27.995 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-07-26T23:35:28.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.008+0000 7f42537fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f425414d1e0 0x7f425414f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.008+0000 7f290effd640 1 -- 172.21.15.139:0/1691464336 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f2910061180 con 0x7f292006e720 2024-07-26T23:35:28.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.008+0000 7f290effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.008+0000 7f290effd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:28.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.008+0000 7f23abfff640 1 -- 172.21.15.139:0/3390344634 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f23a404c060 con 0x7f23b41470c0 2024-07-26T23:35:28.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f42537fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f425414d1e0 0x7f425414f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41478/0 (socket says 172.21.15.139:41478) 2024-07-26T23:35:28.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f42537fe640 1 -- 172.21.15.139:0/2020956911 learned_addr learned my addr 172.21.15.139:0/2020956911 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:28.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f42537fe640 10 monclient(hunting): get_auth_request con 0x7f425414d1e0 auth_method 0 2024-07-26T23:35:28.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f42537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:28.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f42537fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:28.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f42537fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:28.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f23abfff640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f23940789a0 0x7f239407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f23b37fe640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f23940789a0 0x7f239407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.009+0000 7f23b37fe640 10 monclient: get_auth_request con 0x7f23940789a0 auth_method 0 2024-07-26T23:35:28.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.010+0000 7f23b37fe640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f23940789a0 0x7f239407ae60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00ab90 tx=0x7f23ac009250 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.010+0000 7f23abfff640 10 client.4331.objecter ms_handle_connect 0x7f23940789a0 2024-07-26T23:35:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.010+0000 7f42537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.010+0000 7f42537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.011+0000 7f42537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.011+0000 7f7944ff9640 1 -- 172.21.15.139:0/1611537543 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7938065f40 con 0x7f7948068740 2024-07-26T23:35:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.011+0000 7f7944ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-07-26T23:35:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.011+0000 7f7944ff9640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:28.013 INFO:teuthology.orchestra.run.smithi139.stdout:85899345931 2024-07-26T23:35:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.013+0000 7f23abfff640 1 -- 172.21.15.139:0/3390344634 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f23a407f050 con 0x7f23b41470c0 2024-07-26T23:35:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.013+0000 7f23abfff640 10 client.4331.objecter ms_dispatch 0x7f23b4000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.013+0000 7f23abfff640 3 client.4331.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.013+0000 7f23abfff640 3 client.4331.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:28.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.013+0000 7f23abfff640 20 client.4331.objecter dump_active .. 0 homeless 2024-07-26T23:35:28.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 10 librados: watch_flush enter 2024-07-26T23:35:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 10 librados: watch_flush exit 2024-07-26T23:35:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 20 client.4372.objecter shutdown clearing up homeless session... 2024-07-26T23:35:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 10 client.4372.objecter successfully canceled tick 2024-07-26T23:35:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f79340789f0 msgr2=0x7f793407aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f79340789f0 0x7f793407aeb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f793c02b8c0 tx=0x7f793c03d040 comp rx=0 tx=0).stop 2024-07-26T23:35:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 10 monclient: shutdown 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7948068740 msgr2=0x7f794813ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7948068740 0x7f794813ee60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f793800e8a0 tx=0x7f793800ed60 comp rx=0 tx=0).stop 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 shutdown_connections 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f79340789f0 0x7f793407aeb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f794806fa60 0x7f794813e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7948143910 0x7f79480b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 --2- 172.21.15.139:0/1611537543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7948068740 0x7f794813ee60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.014+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 >> 172.21.15.139:0/1611537543 conn(0x7f794806c720 msgr2=0x7f7948137ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 shutdown_connections 2024-07-26T23:35:28.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f794e2f2640 1 -- 172.21.15.139:0/1611537543 wait complete. 2024-07-26T23:35:28.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f794e2f2640 1 librados: shutdown 2024-07-26T23:35:28.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2024-07-26T23:35:28.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 1 -- 172.21.15.139:0/2020956911 >> v1:172.21.15.177:6789/0 conn(0x7f42540b41c0 legacy=0x7f42540b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 1 -- 172.21.15.139:0/2020956911 >> v1:172.21.15.139:6789/0 conn(0x7f42541470c0 legacy=0x7f4254140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:28.018 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:28.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 1 monclient: found mon.noname-c 2024-07-26T23:35:28.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:28.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.015+0000 7f42537fe640 1 -- 172.21.15.139:0/2020956911 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42541474a0 con 0x7f425414d1e0 2024-07-26T23:35:28.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.017+0000 7fd8b77fe640 1 -- 172.21.15.139:0/832341492 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fd8c804d030 con 0x7fd8d014d1e0 2024-07-26T23:35:28.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.017+0000 7fd8b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.019 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.017+0000 7fd8b77fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:28.022 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.021+0000 7f42537fe640 1 --2- 172.21.15.139:0/2020956911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f425414d1e0 0x7f425414f5d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f424c009e50 tx=0x7f424c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dd65cd57178c888 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 1 -- 172.21.15.139:0/2020956911 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c02ca70 con 0x7f425414d1e0 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 10 monclient: dump: 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.026 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f42527fc640 1 -- 172.21.15.139:0/2020956911 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f424c02cc10 con 0x7f425414d1e0 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.024+0000 7f4259457640 5 monclient: authenticate success, global_id 4381 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f4259457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f42527fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:28.027 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f4259457640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f4259457640 4 set_mon_vals no callback set 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f42527fc640 1 -- 172.21.15.139:0/2020956911 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c0343f0 con 0x7f425414d1e0 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f42527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f42527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.025+0000 7f42527fc640 10 monclient: dump: 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.028 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 10 monclient: shutdown 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 1 -- 172.21.15.139:0/2020956911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f425414d1e0 msgr2=0x7f425414f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 1 --2- 172.21.15.139:0/2020956911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f425414d1e0 0x7f425414f5d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f424c009e50 tx=0x7f424c02bed0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 1 -- 172.21.15.139:0/2020956911 shutdown_connections 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 1 --2- 172.21.15.139:0/2020956911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f425414d1e0 0x7f425414f5d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.029 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 1 -- 172.21.15.139:0/2020956911 >> 172.21.15.139:0/2020956911 conn(0x7f42540b62c0 msgr2=0x7f42540b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.030 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 1 -- 172.21.15.139:0/2020956911 shutdown_connections 2024-07-26T23:35:28.030 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.026+0000 7f4259457640 1 -- 172.21.15.139:0/2020956911 wait complete. 2024-07-26T23:35:28.030 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.027+0000 7f4259457640 10 monclient: build_initial_monmap 2024-07-26T23:35:28.030 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.027+0000 7f4259457640 10 monclient: monmap: 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.028+0000 7f4259457640 1 librados: starting msgr at 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.028+0000 7f4259457640 1 librados: starting objecter 2024-07-26T23:35:28.031 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.028+0000 7f4259457640 1 Processor -- start 2024-07-26T23:35:28.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.028+0000 7f4259457640 1 -- start start 2024-07-26T23:35:28.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.028+0000 7f4259457640 1 librados: setting wanted keys 2024-07-26T23:35:28.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.028+0000 7f4259457640 1 librados: calling monclient init 2024-07-26T23:35:28.032 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.029+0000 7f4259457640 10 monclient: init 2024-07-26T23:35:28.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:28.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:35:28.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f42540b41c0 0x7f42540ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 10 monclient(hunting): picked mon.noname-c con 0x7f42540b41c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541470c0 0x7f425413f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 10 monclient(hunting): picked mon.noname-b con 0x7f42541470c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f425414d1e0 0x7f425413ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.030+0000 7f4259457640 10 monclient(hunting): picked mon.noname-a con 0x7f425414d1e0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4254141410 con 0x7f425414d1e0 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4254074850 con 0x7f42541470c0 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f4254141060 con 0x7f42540b41c0 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f424bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541470c0 0x7f425413f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.035 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f424bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541470c0 0x7f425413f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41486/0 (socket says 172.21.15.139:41486) 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f424bfff640 1 -- 172.21.15.139:0/644158293 learned_addr learned my addr 172.21.15.139:0/644158293 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f4259457640 10 monclient(hunting): authenticate will time out at 1361.067737s 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f424bfff640 10 monclient(hunting): get_auth_request con 0x7f42541470c0 auth_method 0 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f424bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f424bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.031+0000 7f424bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.032+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.032+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:28.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.032+0000 7f424bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:28.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345931 got 85899345931 for osd.3 2024-07-26T23:35:28.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-07-26T23:35:28.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 1 -- 172.21.15.139:0/644158293 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f42540b41c0 msgr2=0x7f42540ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f42540b41c0 0x7f42540ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 1 -- 172.21.15.139:0/644158293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f425414d1e0 msgr2=0x7f425413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f425414d1e0 0x7f425413ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:28.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 1 monclient: found mon.noname-b 2024-07-26T23:35:28.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:28.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.035+0000 7f424bfff640 1 -- 172.21.15.139:0/644158293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42540aefc0 con 0x7f42541470c0 2024-07-26T23:35:28.038 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:28.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.037+0000 7f424bfff640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541470c0 0x7f425413f9d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c450 tx=0x7f423c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.037+0000 7f424b7fe640 10 client.?.objecter ms_handle_connect 0x7f42541470c0 2024-07-26T23:35:28.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.037+0000 7f424b7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 1 -- 172.21.15.139:0/644158293 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c0052d0 con 0x7f42541470c0 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: dump: 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.043 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: _finish_auth 0 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:35:28.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 1 -- 172.21.15.139:0/644158293 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f423c005470 con 0x7f42541470c0 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 1 -- 172.21.15.139:0/644158293 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c012870 con 0x7f42541470c0 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f424b7fe640 10 monclient: dump: 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.045 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.046 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f42517fa640 4 set_mon_vals no callback set 2024-07-26T23:35:28.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 5 monclient: authenticate success, global_id 4384 2024-07-26T23:35:28.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 10 monclient: _renew_subs 2024-07-26T23:35:28.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.048 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42540af2a0 con 0x7f42541470c0 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 10 monclient: _renew_subs 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42540addd0 con 0x7f42541470c0 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.041+0000 7f4259457640 1 librados: init done 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.043+0000 7f4259457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.043+0000 7f4259457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.043+0000 7f4259457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.049 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.043+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42540008d0 con 0x7f42541470c0 2024-07-26T23:35:28.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.106+0000 7f6107849640 10 monclient: get_monmap_and_config 2024-07-26T23:35:28.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.106+0000 7f6107849640 10 monclient: build_initial_monmap 2024-07-26T23:35:28.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.106+0000 7f6107849640 10 monclient: monmap: 2024-07-26T23:35:28.108 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:28.107241+0000 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:28.107241+0000 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.106+0000 7f6107849640 1 Processor -- start 2024-07-26T23:35:28.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 1 -- start start 2024-07-26T23:35:28.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient: init 2024-07-26T23:35:28.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:28.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f610006e720 0x7f6100139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient(hunting): picked mon.noname-c con 0x7f610006e720 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6100143d00 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f6100144cd0 0x7f6100145100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient(hunting): picked mon.noname-a con 0x7f6100144cd0 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61000ac810 con 0x7f6100143d00 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f610006eb00 con 0x7f6100144cd0 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f6100144710 con 0x7f610006e720 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.107+0000 7f6107849640 10 monclient(hunting): authenticate will time out at 1361.143999s 2024-07-26T23:35:28.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.108+0000 7f6105de6640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f6100144cd0 0x7f6100145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.109+0000 7f6105de6640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f6100144cd0 0x7f6100145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:32918/0 (socket says 172.21.15.139:32918) 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.109+0000 7f6105de6640 1 -- 172.21.15.139:0/2437980249 learned_addr learned my addr 172.21.15.139:0/2437980249 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.109+0000 7f6105de6640 10 monclient(hunting): get_auth_request con 0x7f6100144cd0 auth_method 0 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.109+0000 7f6105de6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.109+0000 7f6105de6640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.109+0000 7f6105de6640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.110+0000 7f6105de6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.110+0000 7f6105de6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:28.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.110+0000 7f6105de6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:28.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2024-07-26T23:35:28.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 1 -- 172.21.15.139:0/2437980249 >> v2:172.21.15.177:3300/0 conn(0x7f610006e720 msgr2=0x7f6100139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:28.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 1 --2- 172.21.15.139:0/2437980249 >> v2:172.21.15.177:3300/0 conn(0x7f610006e720 0x7f6100139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 1 -- 172.21.15.139:0/2437980249 >> v1:172.21.15.177:6789/0 conn(0x7f6100143d00 legacy=0x7f610013a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:28.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 10 monclient: _finish_hunting 0 2024-07-26T23:35:28.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 1 monclient: found mon.noname-a 2024-07-26T23:35:28.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:28.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.112+0000 7f6105de6640 1 -- 172.21.15.139:0/2437980249 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61000acaf0 con 0x7f6100144cd0 2024-07-26T23:35:28.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.115+0000 7f6105de6640 1 --2- 172.21.15.139:0/2437980249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100144cd0 0x7f6100145100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f61000b7360 tx=0x7f60f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb55e3999a76a8b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.122 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 1 -- 172.21.15.139:0/2437980249 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f802ca80 con 0x7f6100144cd0 2024-07-26T23:35:28.122 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.122 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:28.122 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 10 monclient: dump: 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.123 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.124 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.124 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.124 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f60f7fff640 1 -- 172.21.15.139:0/2437980249 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f802cc20 con 0x7f6100144cd0 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f6107849640 5 monclient: authenticate success, global_id 4377 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.120+0000 7f6107849640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f60f7fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f60f7fff640 1 -- 172.21.15.139:0/2437980249 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f80343f0 con 0x7f6100144cd0 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f60f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f60f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f60f7fff640 10 monclient: dump: 2024-07-26T23:35:28.125 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f6107849640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:28.126 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f6107849640 4 set_mon_vals no callback set 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f6107849640 10 monclient: shutdown 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f6107849640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f6107849640 1 -- 172.21.15.139:0/2437980249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100144cd0 msgr2=0x7f6100145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.121+0000 7f6107849640 1 --2- 172.21.15.139:0/2437980249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100144cd0 0x7f6100145100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f61000b7360 tx=0x7f60f802bee0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.122+0000 7f6107849640 1 -- 172.21.15.139:0/2437980249 shutdown_connections 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.122+0000 7f6107849640 1 --2- 172.21.15.139:0/2437980249 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100144cd0 0x7f6100145100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.122+0000 7f6107849640 1 --2- 172.21.15.139:0/2437980249 >> v2:172.21.15.177:3300/0 conn(0x7f610006e720 0x7f6100139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.122+0000 7f6107849640 1 -- 172.21.15.139:0/2437980249 >> 172.21.15.139:0/2437980249 conn(0x7f61000b9c10 msgr2=0x7f61000ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.132 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.123+0000 7f6107849640 1 -- 172.21.15.139:0/2437980249 shutdown_connections 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.123+0000 7f6107849640 1 -- 172.21.15.139:0/2437980249 wait complete. 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.124+0000 7f6107849640 10 monclient: build_initial_monmap 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.124+0000 7f6107849640 10 monclient: monmap: 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:28.133 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.124+0000 7f6107849640 1 librados: starting msgr at 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.124+0000 7f6107849640 1 librados: starting objecter 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.124+0000 7f6107849640 1 Processor -- start 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.125+0000 7f6107849640 1 -- start start 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.125+0000 7f6107849640 1 librados: setting wanted keys 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.125+0000 7f6107849640 1 librados: calling monclient init 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.125+0000 7f6107849640 10 monclient: init 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f610006e720 0x7f6100140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient(hunting): picked mon.noname-c con 0x7f610006e720 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100143d00 0x7f610013de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient(hunting): picked mon.noname-a con 0x7f6100143d00 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6100144cd0 0x7f610013e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient(hunting): picked mon.noname-b con 0x7f6100144cd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6100145640 con 0x7f6100143d00 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6100141390 con 0x7f6100144cd0 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6107849640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f61001457c0 con 0x7f610006e720 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.126+0000 7f6104de4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100143d00 0x7f610013de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6107849640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6107849640 10 monclient(hunting): authenticate will time out at 1361.163148s 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6104de4640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100143d00 0x7f610013de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:32922/0 (socket says 172.21.15.139:32922) 2024-07-26T23:35:28.135 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6104de4640 1 -- 172.21.15.139:0/2184292383 learned_addr learned my addr 172.21.15.139:0/2184292383 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:28.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6104de4640 10 monclient(hunting): get_auth_request con 0x7f6100143d00 auth_method 0 2024-07-26T23:35:28.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6104de4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:28.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6104de4640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:28.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.127+0000 7f6104de4640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:28.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.128+0000 7f6104de4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:28.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.128+0000 7f6104de4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:28.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.128+0000 7f6104de4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:28.137 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-07-26T23:35:28.137 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 1 -- 172.21.15.139:0/2184292383 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f610006e720 msgr2=0x7f6100140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f610006e720 0x7f6100140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 1 -- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6100144cd0 msgr2=0x7f610013e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6100144cd0 0x7f610013e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 10 monclient: _finish_hunting 0 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 1 monclient: found mon.noname-a 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.132+0000 7f6104de4640 1 -- 172.21.15.139:0/2184292383 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610013d3b0 con 0x7f6100143d00 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.136+0000 7f6104de4640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100143d00 0x7f610013de10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c410 tx=0x7f60f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.136+0000 7f60f5ffb640 10 client.?.objecter ms_handle_connect 0x7f6100143d00 2024-07-26T23:35:28.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.136+0000 7f60f5ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 1 -- 172.21.15.139:0/2184292383 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0017070 con 0x7f6100143d00 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: dump: 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.144 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: _finish_auth 0 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 1 -- 172.21.15.139:0/2184292383 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f00035f0 con 0x7f6100143d00 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 1 -- 172.21.15.139:0/2184292383 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0003a20 con 0x7f6100143d00 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:28.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f5ffb640 10 monclient: dump: 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.140+0000 7f60f77fe640 4 set_mon_vals no callback set 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 5 monclient: authenticate success, global_id 4380 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 10 monclient: _renew_subs 2024-07-26T23:35:28.146 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610013fbf0 con 0x7f6100143d00 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 10 monclient: _renew_subs 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6100140ba0 con 0x7f6100143d00 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.141+0000 7f6107849640 1 librados: init done 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.142+0000 7f6107849640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.142+0000 7f6107849640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.142+0000 7f6107849640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.147 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.142+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610013f970 con 0x7f6100143d00 2024-07-26T23:35:28.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.161+0000 7f5e2a7b9640 10 monclient: get_monmap_and_config 2024-07-26T23:35:28.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.161+0000 7f5e2a7b9640 10 monclient: build_initial_monmap 2024-07-26T23:35:28.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.161+0000 7f5e2a7b9640 10 monclient: monmap: 2024-07-26T23:35:28.162 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:28.162224+0000 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:28.162224+0000 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:28.163 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.161+0000 7f5e2a7b9640 1 Processor -- start 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 1 -- start start 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient: init 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f5e2406e720 0x7f5e24139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient(hunting): picked mon.noname-a con 0x7f5e2406e720 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5e24143d00 0x7f5e2413a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e24143d00 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e24144cd0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e24145830 con 0x7f5e24144cd0 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f5e2406eb00 con 0x7f5e2406e720 2024-07-26T23:35:28.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f5e24144710 con 0x7f5e24143d00 2024-07-26T23:35:28.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:28.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.162+0000 7f5e2a7b9640 10 monclient(hunting): authenticate will time out at 1361.198784s 2024-07-26T23:35:28.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.178+0000 7f60f5ffb640 1 -- 172.21.15.139:0/2184292383 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f60f001a070 con 0x7f6100143d00 2024-07-26T23:35:28.180 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.178+0000 7f60f5ffb640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f60e4078a20 0x7f60e407aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.180 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.179+0000 7f61055e5640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f60e4078a20 0x7f60e407aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.180 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.179+0000 7f61055e5640 10 monclient: get_auth_request con 0x7f60e4078a20 auth_method 0 2024-07-26T23:35:28.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.179+0000 7f61055e5640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f60e4078a20 0x7f60e407aee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009710 tx=0x7f60fc006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.179+0000 7f60f5ffb640 10 client.4380.objecter ms_handle_connect 0x7f60e4078a20 2024-07-26T23:35:28.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f60f5ffb640 1 -- 172.21.15.139:0/2184292383 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f60f005ce90 con 0x7f6100143d00 2024-07-26T23:35:28.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f60f5ffb640 10 client.4380.objecter ms_dispatch 0x7f6100000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:28.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f60f5ffb640 3 client.4380.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:28.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f60f5ffb640 3 client.4380.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:28.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f5e23fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f5e2406e720 0x7f5e24139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f60f5ffb640 20 client.4380.objecter dump_active .. 0 homeless 2024-07-26T23:35:28.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f5e23fff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f5e2406e720 0x7f5e24139bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:32928/0 (socket says 172.21.15.139:32928) 2024-07-26T23:35:28.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f5e23fff640 1 -- 172.21.15.139:0/743155103 learned_addr learned my addr 172.21.15.139:0/743155103 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f5e23fff640 10 monclient(hunting): get_auth_request con 0x7f5e2406e720 auth_method 0 2024-07-26T23:35:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f5e23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f5e23fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.186+0000 7f5e23fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.187+0000 7f5e23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.187+0000 7f5e23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.187+0000 7f5e23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2024-07-26T23:35:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 1 -- 172.21.15.139:0/743155103 >> v2:172.21.15.177:3300/0 conn(0x7f5e24143d00 msgr2=0x7f5e2413a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 1 --2- 172.21.15.139:0/743155103 >> v2:172.21.15.177:3300/0 conn(0x7f5e24143d00 0x7f5e2413a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 1 -- 172.21.15.139:0/743155103 >> v1:172.21.15.139:6789/0 conn(0x7f5e24144cd0 legacy=0x7f5e24145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:28.192 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:28.192 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 1 monclient: found mon.noname-a 2024-07-26T23:35:28.192 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:28.192 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.192 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.190+0000 7f5e23fff640 1 -- 172.21.15.139:0/743155103 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2413a650 con 0x7f5e2406e720 2024-07-26T23:35:28.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.269+0000 7f2924b10640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-26T23:35:28.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.269+0000 7f2924b10640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-26T23:35:28.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.269+0000 7f2924b10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.271 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.269+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f29200ac250 con 0x7f292006e720 2024-07-26T23:35:28.285 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.284+0000 7fd8d50e4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-26T23:35:28.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.284+0000 7fd8d50e4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-26T23:35:28.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.284+0000 7fd8d50e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.284+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd8d00b4ad0 con 0x7fd8d014d1e0 2024-07-26T23:35:28.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.294+0000 7f424b7fe640 1 -- 172.21.15.139:0/644158293 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f423c012a10 con 0x7f42541470c0 2024-07-26T23:35:28.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.295+0000 7f424b7fe640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f42340789f0 0x7f423407aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.295+0000 7f4252ffd640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f42340789f0 0x7f423407aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.295+0000 7f4252ffd640 10 monclient: get_auth_request con 0x7f42340789f0 auth_method 0 2024-07-26T23:35:28.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.296+0000 7f4252ffd640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f42340789f0 0x7f423407aeb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4244001560 tx=0x7f4244007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.296+0000 7f424b7fe640 10 client.4384.objecter ms_handle_connect 0x7f42340789f0 2024-07-26T23:35:28.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.299+0000 7f424b7fe640 1 -- 172.21.15.139:0/644158293 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f423c012d00 con 0x7f42541470c0 2024-07-26T23:35:28.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.299+0000 7f424b7fe640 10 client.4384.objecter ms_dispatch 0x7f4254000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:28.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.299+0000 7f424b7fe640 3 client.4384.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:28.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.299+0000 7f424b7fe640 3 client.4384.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:28.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.299+0000 7f424b7fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-07-26T23:35:28.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.309+0000 7f424b7fe640 1 -- 172.21.15.139:0/644158293 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f423c062070 con 0x7f42541470c0 2024-07-26T23:35:28.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.309+0000 7f424b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.309+0000 7f424b7fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:28.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.315+0000 7f23abfff640 1 -- 172.21.15.139:0/3390344634 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f23a4040070 con 0x7f23b41470c0 2024-07-26T23:35:28.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.315+0000 7f23abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.315+0000 7f23abfff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:28.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.319+0000 7f290effd640 1 -- 172.21.15.139:0/1691464336 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f29100656d0 con 0x7f292006e720 2024-07-26T23:35:28.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.319+0000 7f290effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-07-26T23:35:28.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.319+0000 7f290effd640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:28.321 INFO:teuthology.orchestra.run.smithi139.stdout:90194313226 2024-07-26T23:35:28.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 10 librados: watch_flush enter 2024-07-26T23:35:28.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 10 librados: watch_flush exit 2024-07-26T23:35:28.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 20 client.4378.objecter shutdown clearing up homeless session... 2024-07-26T23:35:28.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 10 client.4378.objecter successfully canceled tick 2024-07-26T23:35:28.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f29000789f0 msgr2=0x7f290007aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f29000789f0 0x7f290007aeb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2914001220 tx=0x7f29140078f0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 10 monclient: shutdown 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f292006e720 msgr2=0x7f29200727d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.321+0000 7f2924b10640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f292006e720 0x7f29200727d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f291000b4a0 tx=0x7f291000b960 comp rx=0 tx=0).stop 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 shutdown_connections 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f29000789f0 0x7f290007aeb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f2920144cd0 0x7f29201e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2920143d00 0x7f29201e0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 --2- 172.21.15.139:0/1691464336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f292006e720 0x7f29200727d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 >> 172.21.15.139:0/1691464336 conn(0x7f29200b9c10 msgr2=0x7f2920138ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 shutdown_connections 2024-07-26T23:35:28.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 -- 172.21.15.139:0/1691464336 wait complete. 2024-07-26T23:35:28.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.322+0000 7f2924b10640 1 librados: shutdown 2024-07-26T23:35:28.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313226 got 90194313226 for osd.2 2024-07-26T23:35:28.335 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:28.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.471+0000 7f60f5ffb640 1 -- 172.21.15.139:0/2184292383 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f610013f970 con 0x7f6100143d00 2024-07-26T23:35:28.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.471+0000 7f60f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.472 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.471+0000 7f60f5ffb640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:28.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.473+0000 7f5e23fff640 1 --2- 172.21.15.139:0/743155103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e2406e720 0x7f5e24139bd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009870 tx=0x7f5e1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd826f30269d172a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.481+0000 7fd8b77fe640 1 -- 172.21.15.139:0/832341492 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd8c80845b0 con 0x7fd8d014d1e0 2024-07-26T23:35:28.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.481+0000 7fd8b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-07-26T23:35:28.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.481+0000 7fd8b77fe640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:28.483 INFO:teuthology.orchestra.run.smithi139.stdout:55834574856 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 10 librados: watch_flush enter 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 10 librados: watch_flush exit 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 20 client.4374.objecter shutdown clearing up homeless session... 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 10 client.4374.objecter successfully canceled tick 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd8bc0789f0 msgr2=0x7fd8bc07aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd8bc0789f0 0x7fd8bc07aeb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0000f60 tx=0x7fd8c00079b0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 10 monclient: shutdown 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.484+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 msgr2=0x7fd8d013fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d013fea0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8032000 tx=0x7fd8c8004a40 comp rx=0 tx=0).stop 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 shutdown_connections 2024-07-26T23:35:28.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd8bc0789f0 0x7fd8bc07aeb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd8d01470c0 0x7fd8d013f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd8d014d1e0 0x7fd8d013fea0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 --2- 172.21.15.139:0/832341492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8d00b41c0 0x7fd8d00add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 >> 172.21.15.139:0/832341492 conn(0x7fd8d00b62c0 msgr2=0x7fd8d01437e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 shutdown_connections 2024-07-26T23:35:28.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 -- 172.21.15.139:0/832341492 wait complete. 2024-07-26T23:35:28.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.485+0000 7fd8d50e4640 1 librados: shutdown 2024-07-26T23:35:28.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e22ffd640 1 -- 172.21.15.139:0/743155103 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14035020 con 0x7f5e2406e720 2024-07-26T23:35:28.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:35:28.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e22ffd640 10 monclient: dump: 2024-07-26T23:35:28.495 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e22ffd640 10 monclient: _finish_auth 0 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e22ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:28.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:28.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e2a7b9640 5 monclient: authenticate success, global_id 4383 2024-07-26T23:35:28.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.493+0000 7f5e2a7b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:28.498 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-07-26T23:35:28.498 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:28.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e22ffd640 1 -- 172.21.15.139:0/743155103 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e14031640 con 0x7f5e2406e720 2024-07-26T23:35:28.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e22ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:28.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e22ffd640 1 -- 172.21.15.139:0/743155103 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1402d040 con 0x7f5e2406e720 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e22ffd640 10 monclient: dump: 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.502 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.503 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e2a7b9640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:28.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.500+0000 7f5e2a7b9640 4 set_mon_vals no callback set 2024-07-26T23:35:28.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 10 monclient: shutdown 2024-07-26T23:35:28.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/743155103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e2406e720 msgr2=0x7f5e24139bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/743155103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e2406e720 0x7f5e24139bd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009870 tx=0x7f5e1402b8f0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/743155103 shutdown_connections 2024-07-26T23:35:28.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/743155103 >> v2:172.21.15.177:3300/0 conn(0x7f5e24143d00 0x7f5e2413a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/743155103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e2406e720 0x7f5e24139bd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/743155103 >> 172.21.15.139:0/743155103 conn(0x7f5e240b9c10 msgr2=0x7f5e240ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/743155103 shutdown_connections 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.501+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/743155103 wait complete. 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.502+0000 7f5e2a7b9640 10 monclient: build_initial_monmap 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.502+0000 7f5e2a7b9640 10 monclient: monmap: 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:28.505 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.506 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:28.506 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:28.506 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:28.506 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.502+0000 7f5e2a7b9640 1 librados: starting msgr at 2024-07-26T23:35:28.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.502+0000 7f5e2a7b9640 1 librados: starting objecter 2024-07-26T23:35:28.506 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.503+0000 7f5e2a7b9640 1 Processor -- start 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.503+0000 7f5e2a7b9640 1 -- start start 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.503+0000 7f5e2a7b9640 1 librados: setting wanted keys 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.503+0000 7f5e2a7b9640 1 librados: calling monclient init 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.503+0000 7f5e2a7b9640 10 monclient: init 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e2406e720 0x7f5e2413d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): picked mon.noname-b con 0x7f5e2406e720 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e24143d00 0x7f5e241d9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): picked mon.noname-a con 0x7f5e24143d00 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5e24144cd0 0x7f5e241d9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.508 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e24144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241441f0 con 0x7f5e24143d00 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240ac8a0 con 0x7f5e2406e720 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f5e24141390 con 0x7f5e24144cd0 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:28.509 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.504+0000 7f5e2a7b9640 10 monclient(hunting): authenticate will time out at 1361.540341s 2024-07-26T23:35:28.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e2406e720 0x7f5e2413d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.511 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e28d56640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5e24144cd0 0x7f5e241d9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e2406e720 0x7f5e2413d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:41506/0 (socket says 172.21.15.139:41506) 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 1 -- 172.21.15.139:0/3670303549 learned_addr learned my addr 172.21.15.139:0/3670303549 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 10 monclient(hunting): get_auth_request con 0x7f5e2406e720 auth_method 0 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e28d56640 10 monclient(hunting): get_auth_request con 0x7f5e24144cd0 auth_method 0 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e28d56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:28.512 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e28d56640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:28.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e28d56640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:28.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:28.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:28.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.505+0000 7f5e23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:28.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.506+0000 7f5e28d56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:28.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.506+0000 7f5e28d56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:28.513 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.506+0000 7f5e28d56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:28.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-07-26T23:35:28.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 1 -- 172.21.15.139:0/3670303549 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5e24144cd0 msgr2=0x7f5e241d9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5e24144cd0 0x7f5e241d9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.514 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 1 -- 172.21.15.139:0/3670303549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e24143d00 msgr2=0x7f5e241d9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e24143d00 0x7f5e241d9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 10 monclient: _finish_hunting 0 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 1 monclient: found mon.noname-b 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.508+0000 7f5e23fff640 1 -- 172.21.15.139:0/3670303549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e241dc4d0 con 0x7f5e2406e720 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.511+0000 7f5e23fff640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e2406e720 0x7f5e2413d0f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5e14000c00 tx=0x7f5e140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.511+0000 7f5e20ff9640 10 client.?.objecter ms_handle_connect 0x7f5e2406e720 2024-07-26T23:35:28.515 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.511+0000 7f5e20ff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:28.541 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.539+0000 7f4259457640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-26T23:35:28.541 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.539+0000 7f4259457640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-26T23:35:28.541 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.539+0000 7f4259457640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.541 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.539+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f42540ae290 con 0x7f42541470c0 2024-07-26T23:35:28.545 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.544+0000 7f23b9bcf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-26T23:35:28.545 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.544+0000 7f23b9bcf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-26T23:35:28.545 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.544+0000 7f23b9bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:35:28.545 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.544+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f23b40b41c0 con 0x7f23b41470c0 2024-07-26T23:35:28.574 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 1 -- 172.21.15.139:0/3670303549 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1403ac60 con 0x7f5e2406e720 2024-07-26T23:35:28.574 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.574 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: dump: 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: _finish_auth 0 2024-07-26T23:35:28.575 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: _check_auth_tickets 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 1 -- 172.21.15.139:0/3670303549 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1403ae00 con 0x7f5e2406e720 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 5 monclient: authenticate success, global_id 4387 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 1 -- 172.21.15.139:0/3670303549 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14044610 con 0x7f5e2406e720 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 10 monclient: _renew_subs 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2413c680 con 0x7f5e2406e720 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:28.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e20ff9640 10 monclient: dump: 2024-07-26T23:35:28.577 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:28.577 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:28.577 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.577 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:28.577 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:28.577 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e227fc640 4 set_mon_vals no callback set 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 10 monclient: _renew_subs 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2413d630 con 0x7f5e2406e720 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.572+0000 7f5e2a7b9640 1 librados: init done 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.573+0000 7f5e2a7b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.573+0000 7f5e2a7b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.573+0000 7f5e2a7b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:28.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.573+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e241451a0 con 0x7f5e2406e720 2024-07-26T23:35:28.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.577+0000 7f424b7fe640 1 -- 172.21.15.139:0/644158293 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f423c06a2b0 con 0x7f42541470c0 2024-07-26T23:35:28.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.577+0000 7f424b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-07-26T23:35:28.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.577+0000 7f424b7fe640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:28.580 INFO:teuthology.orchestra.run.smithi139.stdout:64424509448 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.580+0000 7f4259457640 10 librados: watch_flush enter 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 10 librados: watch_flush exit 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 20 client.4384.objecter shutdown clearing up homeless session... 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 10 client.4384.objecter successfully canceled tick 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f42340789f0 msgr2=0x7f423407aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f42340789f0 0x7f423407aeb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4244001560 tx=0x7f4244007990 comp rx=0 tx=0).stop 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 10 monclient: shutdown 2024-07-26T23:35:28.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541470c0 msgr2=0x7f425413f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541470c0 0x7f425413f9d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c450 tx=0x7f423c00c910 comp rx=0 tx=0).stop 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 shutdown_connections 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f42340789f0 0x7f423407aeb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f425414d1e0 0x7f425413ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42541470c0 0x7f425413f9d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 --2- 172.21.15.139:0/644158293 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f42540b41c0 0x7f42540ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 >> 172.21.15.139:0/644158293 conn(0x7f42540b62c0 msgr2=0x7f42540b73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 shutdown_connections 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 -- 172.21.15.139:0/644158293 wait complete. 2024-07-26T23:35:28.584 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.581+0000 7f4259457640 1 librados: shutdown 2024-07-26T23:35:28.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.7 2024-07-26T23:35:28.597 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:28.618 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.616+0000 7f23abfff640 1 -- 172.21.15.139:0/3390344634 <== mon.2 v2:172.21.15.139:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f23a40834c0 con 0x7f23b41470c0 2024-07-26T23:35:28.618 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.616+0000 7f23abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-07-26T23:35:28.618 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.616+0000 7f23abfff640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:28.619 INFO:teuthology.orchestra.run.smithi139.stdout:90194313229 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.618+0000 7f23b9bcf640 10 librados: watch_flush enter 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.618+0000 7f23b9bcf640 10 librados: watch_flush exit 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.618+0000 7f23b9bcf640 20 client.4331.objecter shutdown clearing up homeless session... 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.618+0000 7f23b9bcf640 10 client.4331.objecter successfully canceled tick 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f23940789a0 msgr2=0x7f239407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f23940789a0 0x7f239407ae60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00ab90 tx=0x7f23ac009250 comp rx=0 tx=0).stop 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 10 monclient: shutdown 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b41470c0 msgr2=0x7f23b4142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b41470c0 0x7f23b4142ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009870 tx=0x7f23a40046c0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 shutdown_connections 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f23940789a0 0x7f239407ae60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f23b41470c0 0x7f23b4142ac0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23b4140e90 0x7f23b413df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 --2- 172.21.15.139:0/3390344634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b40b48d0 0x7f23b413d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 >> 172.21.15.139:0/3390344634 conn(0x7f23b40b62c0 msgr2=0x7f23b40b73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 shutdown_connections 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 -- 172.21.15.139:0/3390344634 wait complete. 2024-07-26T23:35:28.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.619+0000 7f23b9bcf640 1 librados: shutdown 2024-07-26T23:35:28.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313229 got 90194313229 for osd.0 2024-07-26T23:35:28.632 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:28.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.684+0000 7f6107849640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-26T23:35:28.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.684+0000 7f6107849640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-26T23:35:28.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.684+0000 7f6107849640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:35:28.686 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.684+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6100145100 con 0x7f6100143d00 2024-07-26T23:35:28.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.688+0000 7f60f5ffb640 1 -- 172.21.15.139:0/2184292383 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f60f0021020 con 0x7f6100143d00 2024-07-26T23:35:28.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.688+0000 7f60f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-07-26T23:35:28.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.688+0000 7f60f5ffb640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:28.690 INFO:teuthology.orchestra.run.smithi139.stdout:85899345929 2024-07-26T23:35:28.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 10 librados: watch_flush enter 2024-07-26T23:35:28.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 10 librados: watch_flush exit 2024-07-26T23:35:28.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 20 client.4380.objecter shutdown clearing up homeless session... 2024-07-26T23:35:28.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 10 client.4380.objecter successfully canceled tick 2024-07-26T23:35:28.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f60e4078a20 msgr2=0x7f60e407aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f60e4078a20 0x7f60e407aee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009710 tx=0x7f60fc006c90 comp rx=0 tx=0).stop 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 10 monclient: shutdown 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100143d00 msgr2=0x7f610013de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100143d00 0x7f610013de10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c410 tx=0x7f60f000c8d0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 shutdown_connections 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f60e4078a20 0x7f60e407aee0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6100144cd0 0x7f610013e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6100143d00 0x7f610013de10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 --2- 172.21.15.139:0/2184292383 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f610006e720 0x7f6100140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:28.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 >> 172.21.15.139:0/2184292383 conn(0x7f61000b9c10 msgr2=0x7f6100141fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:28.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 shutdown_connections 2024-07-26T23:35:28.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 -- 172.21.15.139:0/2184292383 wait complete. 2024-07-26T23:35:28.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.690+0000 7f6107849640 1 librados: shutdown 2024-07-26T23:35:28.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345929 for osd.1 2024-07-26T23:35:28.703 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:28.828 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.827+0000 7f5e20ff9640 1 -- 172.21.15.139:0/3670303549 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f5e14038020 con 0x7f5e2406e720 2024-07-26T23:35:28.829 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e20ff9640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5e100789f0 0x7f5e1007aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e20ff9640 1 -- 172.21.15.139:0/3670303549 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f5e140baaf0 con 0x7f5e2406e720 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e20ff9640 10 client.4387.objecter ms_dispatch 0x7f5e24000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e20ff9640 3 client.4387.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e20ff9640 3 client.4387.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e20ff9640 20 client.4387.objecter dump_active .. 0 homeless 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e237fe640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5e100789f0 0x7f5e1007aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e237fe640 10 monclient: get_auth_request con 0x7f5e100789f0 auth_method 0 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e237fe640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5e100789f0 0x7f5e1007aeb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e2413bff0 tx=0x7f5e18006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:28.830 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.828+0000 7f5e20ff9640 10 client.4387.objecter ms_handle_connect 0x7f5e100789f0 2024-07-26T23:35:28.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.838+0000 7f5e20ff9640 1 -- 172.21.15.139:0/3670303549 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f5e1403f070 con 0x7f5e2406e720 2024-07-26T23:35:28.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.838+0000 7f5e20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:28.839 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:28.838+0000 7f5e20ff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:29.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.036+0000 7f5e2a7b9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-26T23:35:29.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.036+0000 7f5e2a7b9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-26T23:35:29.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.036+0000 7f5e2a7b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.036+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f5e240ac250 con 0x7f5e2406e720 2024-07-26T23:35:29.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.040+0000 7f5e20ff9640 1 -- 172.21.15.139:0/3670303549 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5e1408c1e0 con 0x7f5e2406e720 2024-07-26T23:35:29.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.040+0000 7f5e20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-07-26T23:35:29.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.040+0000 7f5e20ff9640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:29.041 INFO:teuthology.orchestra.run.smithi139.stdout:55834574856 2024-07-26T23:35:29.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 10 librados: watch_flush enter 2024-07-26T23:35:29.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 10 librados: watch_flush exit 2024-07-26T23:35:29.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 20 client.4387.objecter shutdown clearing up homeless session... 2024-07-26T23:35:29.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 10 client.4387.objecter successfully canceled tick 2024-07-26T23:35:29.044 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5e100789f0 msgr2=0x7f5e1007aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5e100789f0 0x7f5e1007aeb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e2413bff0 tx=0x7f5e18006c90 comp rx=0 tx=0).stop 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 10 monclient: shutdown 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e2406e720 msgr2=0x7f5e2413d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e2406e720 0x7f5e2413d0f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5e14000c00 tx=0x7f5e140046c0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 shutdown_connections 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f5e100789f0 0x7f5e1007aeb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f5e24144cd0 0x7f5e241d9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5e24143d00 0x7f5e241d9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 --2- 172.21.15.139:0/3670303549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e2406e720 0x7f5e2413d0f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 >> 172.21.15.139:0/3670303549 conn(0x7f5e240b9c10 msgr2=0x7f5e240b8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:29.045 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 shutdown_connections 2024-07-26T23:35:29.046 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 -- 172.21.15.139:0/3670303549 wait complete. 2024-07-26T23:35:29.046 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.042+0000 7f5e2a7b9640 1 librados: shutdown 2024-07-26T23:35:29.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-07-26T23:35:29.055 DEBUG:teuthology.parallel:result is None 2024-07-26T23:35:29.055 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T23:35:29.055 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T23:35:29.153 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.152+0000 7f771ce95640 10 monclient: get_monmap_and_config 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.152+0000 7f771ce95640 10 monclient: build_initial_monmap 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.152+0000 7f771ce95640 10 monclient: monmap: 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:29.153336+0000 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:29.153336+0000 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:29.154 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:29.155 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:29.155 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:29.155 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.158 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.152+0000 7f771ce95640 1 Processor -- start 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.152+0000 7f771ce95640 1 -- start start 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.152+0000 7f771ce95640 10 monclient: init 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77180b20e0 0x7f7718140b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient(hunting): picked mon.noname-b con 0x7f77180b20e0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77181410b0 addr v1:172.21.15.139:6789/0 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7718147750 0x7f771814e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient(hunting): picked mon.noname-c con 0x7f7718147750 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77180709b0 con 0x7f77181410b0 2024-07-26T23:35:29.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f77180b24c0 con 0x7f7718147750 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f77181414d0 con 0x7f77180b20e0 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.153+0000 7f771ce95640 10 monclient(hunting): authenticate will time out at 1362.189733s 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f771659c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77180b20e0 0x7f7718140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f7715d9b640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f77181410b0 0x7f7718147040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:33396/0 (socket says 172.21.15.139:33396) 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f7715d9b640 1 -- 172.21.15.139:0/3939940949 learned_addr learned my addr 172.21.15.139:0/3939940949 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f7716d9d640 1 --2- 172.21.15.139:0/3939940949 >> v2:172.21.15.177:3300/0 conn(0x7f7718147750 0x7f771814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f771659c640 10 monclient(hunting): get_auth_request con 0x7f77180b20e0 auth_method 0 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f771659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.160 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f771659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f771659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f7716d9d640 10 monclient(hunting): get_auth_request con 0x7f7718147750 auth_method 0 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f7716d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f7716d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.154+0000 7f7716d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.155+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.155+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.155+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.155+0000 7f7716d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.155+0000 7f7716d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.155+0000 7f7716d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f771559a640 1 -- 172.21.15.139:0/3939940949 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499736851 0 0) 0x7f77180709b0 con 0x7f77181410b0 2024-07-26T23:35:29.161 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f771559a640 10 monclient(hunting): _negotiate 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f771559a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f771559a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f771559a640 10 monclient(hunting): my global_id is 4386 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f771559a640 1 -- 172.21.15.139:0/3939940949 --> v1:172.21.15.139:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7700003520 con 0x7f77181410b0 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 1 -- 172.21.15.139:0/3939940949 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77180b20e0 msgr2=0x7f7718140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 1 --2- 172.21.15.139:0/3939940949 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77180b20e0 0x7f7718140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 1 -- 172.21.15.139:0/3939940949 >> v1:172.21.15.139:6789/0 conn(0x7f77181410b0 legacy=0x7f7718147040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 10 monclient: _finish_hunting 0 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 1 monclient: found mon.noname-c 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.162 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.157+0000 7f7716d9d640 1 -- 172.21.15.139:0/3939940949 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718070c20 con 0x7f7718147750 2024-07-26T23:35:29.163 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.159+0000 7f7716d9d640 1 --2- 172.21.15.139:0/3939940949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7718147750 0x7f771814e190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f77080097a0 tx=0x7f770802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce214ee22101d741 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 1 -- 172.21.15.139:0/3939940949 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708035020 con 0x7f7718147750 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: dump: 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.164 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: _finish_auth 0 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: _check_auth_tickets 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 1 -- 172.21.15.139:0/3939940949 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7708031830 con 0x7f7718147750 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771ce95640 5 monclient: authenticate success, global_id 4390 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771ce95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:29.165 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771ce95640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771ce95640 4 set_mon_vals no callback set 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 1 -- 172.21.15.139:0/3939940949 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708031c00 con 0x7f7718147750 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.162+0000 7f771559a640 10 monclient: dump: 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.166 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.167 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.167 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.167 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.167 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.167 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.170 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 10 monclient: shutdown 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 -- 172.21.15.139:0/3939940949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7718147750 msgr2=0x7f771814e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 --2- 172.21.15.139:0/3939940949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7718147750 0x7f771814e190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f77080097a0 tx=0x7f770802b8d0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 -- 172.21.15.139:0/3939940949 shutdown_connections 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 --2- 172.21.15.139:0/3939940949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7718147750 0x7f771814e190 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 --2- 172.21.15.139:0/3939940949 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77180b20e0 0x7f7718140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 -- 172.21.15.139:0/3939940949 >> 172.21.15.139:0/3939940949 conn(0x7f77180b2f60 msgr2=0x7f77180b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 -- 172.21.15.139:0/3939940949 shutdown_connections 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.163+0000 7f771ce95640 1 -- 172.21.15.139:0/3939940949 wait complete. 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 10 monclient: build_initial_monmap 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 10 monclient: monmap: 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:29.171 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 1 librados: starting msgr at 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 1 librados: starting objecter 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 1 Processor -- start 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 1 -- start start 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 1 librados: setting wanted keys 2024-07-26T23:35:29.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 1 librados: calling monclient init 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.164+0000 7f771ce95640 10 monclient: init 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77180b20e0 0x7f7718072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): picked mon.noname-b con 0x7f77180b20e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77181410b0 0x7f7718071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): picked mon.noname-c con 0x7f77181410b0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7718147750 0x7f7718145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): picked mon.noname-a con 0x7f7718147750 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7718147bb0 con 0x7f7718147750 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f77180afdd0 con 0x7f77180b20e0 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f77180b03c0 con 0x7f77181410b0 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.165+0000 7f771ce95640 10 monclient(hunting): authenticate will time out at 1362.201773s 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7715d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77181410b0 0x7f7718071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7716d9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7718147750 0x7f7718145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f771659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77180b20e0 0x7f7718072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7715d9b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77181410b0 0x7f7718071dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:60932/0 (socket says 172.21.15.139:60932) 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7715d9b640 1 -- 172.21.15.139:0/2381269081 learned_addr learned my addr 172.21.15.139:0/2381269081 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7716d9d640 10 monclient(hunting): get_auth_request con 0x7f7718147750 auth_method 0 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7716d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7716d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7716d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7715d9b640 10 monclient(hunting): get_auth_request con 0x7f77181410b0 auth_method 0 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7715d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7715d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f7715d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f771659c640 10 monclient(hunting): get_auth_request con 0x7f77180b20e0 auth_method 0 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f771659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f771659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.166+0000 7f771659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f7716d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f7716d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f7716d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f7715d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.175 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f7715d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.176 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f7715d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.176 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.176 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.176 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.167+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 1 -- 172.21.15.139:0/2381269081 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77181410b0 msgr2=0x7f7718071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77181410b0 0x7f7718071dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 1 -- 172.21.15.139:0/2381269081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7718147750 msgr2=0x7f7718145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7718147750 0x7f7718145130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 10 monclient: _finish_hunting 0 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 1 monclient: found mon.noname-b 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.178 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.170+0000 7f771659c640 1 -- 172.21.15.139:0/2381269081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718073c70 con 0x7f77180b20e0 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.172+0000 7f771659c640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77180b20e0 0x7f7718072ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f771000e850 tx=0x7f771000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.172+0000 7f7706ffd640 10 client.?.objecter ms_handle_connect 0x7f77180b20e0 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.172+0000 7f7706ffd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.175+0000 7f7706ffd640 1 -- 172.21.15.139:0/2381269081 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710017070 con 0x7f77180b20e0 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: dump: 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.179 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.180 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.180 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.180 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.180 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.180 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.180 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.180 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: _finish_auth 0 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 1 -- 172.21.15.139:0/2381269081 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f771000c940 con 0x7f77180b20e0 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 1 -- 172.21.15.139:0/2381269081 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771000cd10 con 0x7f77180b20e0 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 5 monclient: authenticate success, global_id 4393 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 10 monclient: _renew_subs 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7718073f50 con 0x7f77180b20e0 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7706ffd640 10 monclient: dump: 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f7714d99640 4 set_mon_vals no callback set 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 10 monclient: _renew_subs 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77181d0790 con 0x7f77180b20e0 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.176+0000 7f771ce95640 1 librados: init done 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.177+0000 7f771ce95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.177+0000 7f771ce95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.177+0000 7f771ce95640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.183 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.177+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7718140b70 con 0x7f77180b20e0 2024-07-26T23:35:29.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.429+0000 7f7706ffd640 1 -- 172.21.15.139:0/2381269081 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f771001a070 con 0x7f77180b20e0 2024-07-26T23:35:29.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.429+0000 7f7706ffd640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f76f8078a20 0x7f76f807aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.430+0000 7f7715d9b640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f76f8078a20 0x7f76f807aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.430+0000 7f7715d9b640 10 monclient: get_auth_request con 0x7f76f8078a20 auth_method 0 2024-07-26T23:35:29.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.430+0000 7f7706ffd640 10 client.4393.objecter ms_handle_connect 0x7f76f8078a20 2024-07-26T23:35:29.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.430+0000 7f7715d9b640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f76f8078a20 0x7f76f807aee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f77181cd040 tx=0x7f770c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:29.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.433+0000 7f7706ffd640 1 -- 172.21.15.139:0/2381269081 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f771005dc60 con 0x7f77180b20e0 2024-07-26T23:35:29.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.433+0000 7f7706ffd640 10 client.4393.objecter ms_dispatch 0x7f7718000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:29.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.433+0000 7f7706ffd640 3 client.4393.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:29.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.433+0000 7f7706ffd640 3 client.4393.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:29.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.433+0000 7f7706ffd640 20 client.4393.objecter dump_active .. 0 homeless 2024-07-26T23:35:29.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.444+0000 7f7706ffd640 1 -- 172.21.15.139:0/2381269081 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f77100619d0 con 0x7f77180b20e0 2024-07-26T23:35:29.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.444+0000 7f7706ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:29.445 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.444+0000 7f7706ffd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:29.642 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.640+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f771813e930 con 0x7f76f8078a20 2024-07-26T23:35:29.643 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.642+0000 7f7706ffd640 1 -- 172.21.15.139:0/2381269081 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35842 (secure 0 0 0) 0x7f771813e930 con 0x7f76f8078a20 2024-07-26T23:35:29.644 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:35:29.644 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-07-26T23:35:29.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 10 librados: watch_flush enter 2024-07-26T23:35:29.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 10 librados: watch_flush exit 2024-07-26T23:35:29.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 20 client.4393.objecter shutdown clearing up homeless session... 2024-07-26T23:35:29.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 10 client.4393.objecter successfully canceled tick 2024-07-26T23:35:29.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f76f8078a20 msgr2=0x7f76f807aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f76f8078a20 0x7f76f807aee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f77181cd040 tx=0x7f770c009040 comp rx=0 tx=0).stop 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 10 monclient: shutdown 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77180b20e0 msgr2=0x7f7718072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77180b20e0 0x7f7718072ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f771000e850 tx=0x7f771000ed10 comp rx=0 tx=0).stop 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 shutdown_connections 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f76f8078a20 0x7f76f807aee0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f77181410b0 0x7f7718071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7718147750 0x7f7718145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 --2- 172.21.15.139:0/2381269081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77180b20e0 0x7f7718072ce0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 >> 172.21.15.139:0/2381269081 conn(0x7f77180b2f60 msgr2=0x7f7718143c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 shutdown_connections 2024-07-26T23:35:29.647 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 -- 172.21.15.139:0/2381269081 wait complete. 2024-07-26T23:35:29.648 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.644+0000 7f771ce95640 1 librados: shutdown 2024-07-26T23:35:29.656 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-07-26T23:35:27.817953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219120,"kb_used_data":2936,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755600,"statfs":{"total":773094113280,"available":772869734400,"internally_reserved":0,"allocated":3006464,"data_stored":1605166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001288"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.233345+0000","last_change":"2024-07-26T23:35:22.236663+0000","last_active":"2024-07-26T23:35:22.233345+0000","last_peered":"2024-07-26T23:35:22.233345+0000","last_clean":"2024-07-26T23:35:22.233345+0000","last_became_active":"2024-07-26T23:35:18.556903+0000","last_became_peered":"2024-07-26T23:35:18.556903+0000","last_unstale":"2024-07-26T23:35:22.233345+0000","last_undegraded":"2024-07-26T23:35:22.233345+0000","last_fullsized":"2024-07-26T23:35:22.233345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:27.775888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064553026999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.941067+0000","last_change":"2024-07-26T23:35:22.979978+0000","last_active":"2024-07-26T23:35:22.941067+0000","last_peered":"2024-07-26T23:35:22.941067+0000","last_clean":"2024-07-26T23:35:22.941067+0000","last_became_active":"2024-07-26T23:35:18.587303+0000","last_became_peered":"2024-07-26T23:35:18.587303+0000","last_unstale":"2024-07-26T23:35:22.941067+0000","last_undegraded":"2024-07-26T23:35:22.941067+0000","last_fullsized":"2024-07-26T23:35:22.941067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:51:04.403633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.077704690000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.246462+0000","last_change":"2024-07-26T23:35:22.251048+0000","last_active":"2024-07-26T23:35:22.246462+0000","last_peered":"2024-07-26T23:35:22.246462+0000","last_clean":"2024-07-26T23:35:22.246462+0000","last_became_active":"2024-07-26T23:35:17.925979+0000","last_became_peered":"2024-07-26T23:35:17.925979+0000","last_unstale":"2024-07-26T23:35:22.246462+0000","last_undegraded":"2024-07-26T23:35:22.246462+0000","last_fullsized":"2024-07-26T23:35:22.246462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:20.031998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073846415999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.343325+0000","last_change":"2024-07-26T23:35:22.376891+0000","last_active":"2024-07-26T23:35:22.343325+0000","last_peered":"2024-07-26T23:35:22.343325+0000","last_clean":"2024-07-26T23:35:22.343325+0000","last_became_active":"2024-07-26T23:35:18.507138+0000","last_became_peered":"2024-07-26T23:35:18.507138+0000","last_unstale":"2024-07-26T23:35:22.343325+0000","last_undegraded":"2024-07-26T23:35:22.343325+0000","last_fullsized":"2024-07-26T23:35:22.343325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:56:22.382897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047472209000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.220704+0000","last_change":"2024-07-26T23:35:22.220704+0000","last_active":"2024-07-26T23:35:22.220704+0000","last_peered":"2024-07-26T23:35:22.220704+0000","last_clean":"2024-07-26T23:35:22.220704+0000","last_became_active":"2024-07-26T23:35:18.492082+0000","last_became_peered":"2024-07-26T23:35:18.492082+0000","last_unstale":"2024-07-26T23:35:22.220704+0000","last_undegraded":"2024-07-26T23:35:22.220704+0000","last_fullsized":"2024-07-26T23:35:22.220704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:00.865495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098681633000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.055533+0000","last_change":"2024-07-26T23:35:22.093863+0000","last_active":"2024-07-26T23:35:22.055533+0000","last_peered":"2024-07-26T23:35:22.055533+0000","last_clean":"2024-07-26T23:35:22.055533+0000","last_became_active":"2024-07-26T23:35:17.814359+0000","last_became_peered":"2024-07-26T23:35:17.814359+0000","last_unstale":"2024-07-26T23:35:22.055533+0000","last_undegraded":"2024-07-26T23:35:22.055533+0000","last_fullsized":"2024-07-26T23:35:22.055533+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:27:36.663020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060284956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.156536+0000","last_change":"2024-07-26T23:35:22.212052+0000","last_active":"2024-07-26T23:35:22.156536+0000","last_peered":"2024-07-26T23:35:22.156536+0000","last_clean":"2024-07-26T23:35:22.156536+0000","last_became_active":"2024-07-26T23:35:17.846505+0000","last_became_peered":"2024-07-26T23:35:17.846505+0000","last_unstale":"2024-07-26T23:35:22.156536+0000","last_undegraded":"2024-07-26T23:35:22.156536+0000","last_fullsized":"2024-07-26T23:35:22.156536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:31.369623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11625252899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.230355+0000","last_change":"2024-07-26T23:35:22.241392+0000","last_active":"2024-07-26T23:35:22.230355+0000","last_peered":"2024-07-26T23:35:22.230355+0000","last_clean":"2024-07-26T23:35:22.230355+0000","last_became_active":"2024-07-26T23:35:19.087174+0000","last_became_peered":"2024-07-26T23:35:19.087174+0000","last_unstale":"2024-07-26T23:35:22.230355+0000","last_undegraded":"2024-07-26T23:35:22.230355+0000","last_fullsized":"2024-07-26T23:35:22.230355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:54:22.975250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.074002362000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'32","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:23.723611+0000","last_change":"2024-07-26T23:35:17.404909+0000","last_active":"2024-07-26T23:35:23.723611+0000","last_peered":"2024-07-26T23:35:23.723611+0000","last_clean":"2024-07-26T23:35:23.723611+0000","last_became_active":"2024-07-26T23:35:16.707863+0000","last_became_peered":"2024-07-26T23:35:16.707863+0000","last_unstale":"2024-07-26T23:35:23.723611+0000","last_undegraded":"2024-07-26T23:35:23.723611+0000","last_fullsized":"2024-07-26T23:35:23.723611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_clean_scrub_stamp":"2024-07-26T23:35:01.337925+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:50:39.235610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313229,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":20,"seq":85899345931,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":101200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":20,"seq":85899345929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":499021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":101200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":675840,"data_stored":499021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T23:35:29.658 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T23:35:29.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.756+0000 7fb7e5e49640 10 monclient: get_monmap_and_config 2024-07-26T23:35:29.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.756+0000 7fb7e5e49640 10 monclient: build_initial_monmap 2024-07-26T23:35:29.758 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.756+0000 7fb7e5e49640 10 monclient: monmap: 2024-07-26T23:35:29.758 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:29.758 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.758 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:29.757742+0000 2024-07-26T23:35:29.758 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:29.757742+0000 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:29.759 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.757+0000 7fb7e5e49640 1 Processor -- start 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.757+0000 7fb7e5e49640 1 -- start start 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.757+0000 7fb7e5e49640 10 monclient: init 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7e006e2f0 0x7fb7e0140c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): picked mon.noname-c con 0x7fb7e006e2f0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0141150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e0074a60 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:29.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb7e006f390 0x7fb7e00b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): picked mon.noname-a con 0x7fb7e006f390 addr v2:172.21.15.139:3300/0 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fb7e006e6d0 con 0x7fb7e006f390 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb7e006ec60 con 0x7fb7e006e2f0 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e00b0190 con 0x7fb7e0074a60 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7e5e49640 10 monclient(hunting): authenticate will time out at 1362.794227s 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7df7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7e006e2f0 0x7fb7e0140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7deffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.758+0000 7fb7dffff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fb7e006f390 0x7fb7e00b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7e006e2f0 0x7fb7e0140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:39916/0 (socket says 172.21.15.139:39916) 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 1 -- 172.21.15.139:0/4251086234 learned_addr learned my addr 172.21.15.139:0/4251086234 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 10 monclient(hunting): get_auth_request con 0x7fb7e006e2f0 auth_method 0 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7dffff640 10 monclient(hunting): get_auth_request con 0x7fb7e006f390 auth_method 0 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7dffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7dffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.764 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7deffd640 10 monclient(hunting): get_auth_request con 0x7fb7e0074a60 auth_method 0 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7deffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.765 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.759+0000 7fb7deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2024-07-26T23:35:29.766 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 1 -- 172.21.15.139:0/4251086234 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 msgr2=0x7fb7e0141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 1 --2- 172.21.15.139:0/4251086234 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 1 -- 172.21.15.139:0/4251086234 >> v2:172.21.15.139:3300/0 conn(0x7fb7e006f390 msgr2=0x7fb7e00b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 1 --2- 172.21.15.139:0/4251086234 >> v2:172.21.15.139:3300/0 conn(0x7fb7e006f390 0x7fb7e00b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 10 monclient: _finish_hunting 0 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 1 monclient: found mon.noname-c 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.761+0000 7fb7df7fe640 1 -- 172.21.15.139:0/4251086234 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e00b0470 con 0x7fb7e006e2f0 2024-07-26T23:35:29.767 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.763+0000 7fb7df7fe640 1 --2- 172.21.15.139:0/4251086234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006e2f0 0x7fb7e0140c10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00b8730 tx=0x7fb7d002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99328b2f25d7f2eb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:29.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 1 -- 172.21.15.139:0/4251086234 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0038070 con 0x7fb7e006e2f0 2024-07-26T23:35:29.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:35:29.768 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: dump: 2024-07-26T23:35:29.768 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.768 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 1 -- 172.21.15.139:0/4251086234 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7d002f050 con 0x7fb7e006e2f0 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 5 monclient: authenticate success, global_id 4396 2024-07-26T23:35:29.769 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 1 -- 172.21.15.139:0/4251086234 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0031970 con 0x7fb7e006e2f0 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 4 set_mon_vals no callback set 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7de7fc640 10 monclient: dump: 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.770 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.771 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.771 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.771 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.771 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.771 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.775 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 10 monclient: shutdown 2024-07-26T23:35:29.775 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 -- 172.21.15.139:0/4251086234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006e2f0 msgr2=0x7fb7e0140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/4251086234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006e2f0 0x7fb7e0140c10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00b8730 tx=0x7fb7d002b7f0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 -- 172.21.15.139:0/4251086234 shutdown_connections 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/4251086234 >> v2:172.21.15.139:3300/0 conn(0x7fb7e006f390 0x7fb7e00b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/4251086234 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/4251086234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006e2f0 0x7fb7e0140c10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 -- 172.21.15.139:0/4251086234 >> 172.21.15.139:0/4251086234 conn(0x7fb7e00b7ac0 msgr2=0x7fb7e00b7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 -- 172.21.15.139:0/4251086234 shutdown_connections 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.767+0000 7fb7e5e49640 1 -- 172.21.15.139:0/4251086234 wait complete. 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.768+0000 7fb7e5e49640 10 monclient: build_initial_monmap 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.768+0000 7fb7e5e49640 10 monclient: monmap: 2024-07-26T23:35:29.776 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.768+0000 7fb7e5e49640 1 librados: starting msgr at 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.768+0000 7fb7e5e49640 1 librados: starting objecter 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.768+0000 7fb7e5e49640 1 Processor -- start 2024-07-26T23:35:29.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 -- start start 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 librados: setting wanted keys 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 librados: calling monclient init 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient: init 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7e006e2f0 0x7fb7e0141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): picked mon.noname-a con 0x7fb7e006e2f0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006f390 0x7fb7e0142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e006f390 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): picked mon.noname-c con 0x7fb7e0074a60 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.778 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e00b1230 con 0x7fb7e006e2f0 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e00b10b0 con 0x7fb7e006f390 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e00b13b0 con 0x7fb7e0074a60 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.769+0000 7fb7e5e49640 10 monclient(hunting): authenticate will time out at 1362.805722s 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7deffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006f390 0x7fb7e0142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7dffff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7df7fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7e006e2f0 0x7fb7e0141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7deffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006f390 0x7fb7e0142030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:39918/0 (socket says 172.21.15.139:39918) 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7deffd640 1 -- 172.21.15.139:0/3775305053 learned_addr learned my addr 172.21.15.139:0/3775305053 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7deffd640 10 monclient(hunting): get_auth_request con 0x7fb7e006f390 auth_method 0 2024-07-26T23:35:29.779 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7deffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7dffff640 10 monclient(hunting): get_auth_request con 0x7fb7e0074a60 auth_method 0 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7dffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7dffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7df7fe640 10 monclient(hunting): get_auth_request con 0x7fb7e006e2f0 auth_method 0 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7df7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.770+0000 7fb7df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.771+0000 7fb7df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 10 monclient(hunting): handle_auth_done global_id 4399 payload 824 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 1 -- 172.21.15.139:0/3775305053 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 msgr2=0x7fb7e0143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 1 -- 172.21.15.139:0/3775305053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7e006e2f0 msgr2=0x7fb7e0141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:29.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7e006e2f0 0x7fb7e0141720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:29.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 10 monclient: _finish_hunting 0 2024-07-26T23:35:29.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 1 monclient: found mon.noname-b 2024-07-26T23:35:29.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:29.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.773+0000 7fb7deffd640 1 -- 172.21.15.139:0/3775305053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e0142570 con 0x7fb7e006f390 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.775+0000 7fb7deffd640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006f390 0x7fb7e0142030 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400e8a0 tx=0x7fb7d400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.776+0000 7fb7c7fff640 10 client.?.objecter ms_handle_connect 0x7fb7e006f390 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.776+0000 7fb7c7fff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 1 -- 172.21.15.139:0/3775305053 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4017070 con 0x7fb7e006f390 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: dump: 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.784 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: _finish_auth 0 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: _check_auth_tickets 2024-07-26T23:35:29.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 5 monclient: authenticate success, global_id 4399 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 1 -- 172.21.15.139:0/3775305053 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7d400c940 con 0x7fb7e006f390 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 10 monclient: _renew_subs 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 1 -- 172.21.15.139:0/3775305053 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d400cd30 con 0x7fb7e006f390 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e0143040 con 0x7fb7e006f390 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7c7fff640 10 monclient: dump: 2024-07-26T23:35:29.786 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:29.787 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7ddffb640 4 set_mon_vals no callback set 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 10 monclient: _renew_subs 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.779+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e0143600 con 0x7fb7e006f390 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.780+0000 7fb7e5e49640 1 librados: init done 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.780+0000 7fb7e5e49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.780+0000 7fb7e5e49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.780+0000 7fb7e5e49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:29.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:29.780+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e00b0b70 con 0x7fb7e006f390 2024-07-26T23:35:30.036 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.034+0000 7fb7c7fff640 1 -- 172.21.15.139:0/3775305053 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fb7d401a070 con 0x7fb7e006f390 2024-07-26T23:35:30.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7c7fff640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb7cc0789f0 0x7fb7cc07aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:30.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7c7fff640 1 -- 172.21.15.139:0/3775305053 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7fb7d4098a20 con 0x7fb7e006f390 2024-07-26T23:35:30.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7c7fff640 10 client.4399.objecter ms_dispatch 0x7fb7e0000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:30.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7c7fff640 3 client.4399.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:30.037 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7c7fff640 3 client.4399.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:30.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7c7fff640 20 client.4399.objecter dump_active .. 0 homeless 2024-07-26T23:35:30.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7df7fe640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb7cc0789f0 0x7fb7cc07aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:30.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.035+0000 7fb7df7fe640 10 monclient: get_auth_request con 0x7fb7cc0789f0 auth_method 0 2024-07-26T23:35:30.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.036+0000 7fb7c7fff640 10 client.4399.objecter ms_handle_connect 0x7fb7cc0789f0 2024-07-26T23:35:30.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.036+0000 7fb7df7fe640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb7cc0789f0 0x7fb7cc07aeb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00b8730 tx=0x7fb7d0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:30.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.045+0000 7fb7c7fff640 1 -- 172.21.15.139:0/3775305053 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fb7d4061880 con 0x7fb7e006f390 2024-07-26T23:35:30.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.045+0000 7fb7c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:30.047 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.045+0000 7fb7c7fff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:30.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.232+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7e01439a0 con 0x7fb7cc0789f0 2024-07-26T23:35:30.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.233+0000 7fb7c7fff640 1 -- 172.21.15.139:0/3775305053 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35818 (secure 0 0 0) 0x7fb7e01439a0 con 0x7fb7cc0789f0 2024-07-26T23:35:30.235 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:35:30.235 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.235+0000 7fb7e5e49640 10 librados: watch_flush enter 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.235+0000 7fb7e5e49640 10 librados: watch_flush exit 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.235+0000 7fb7e5e49640 20 client.4399.objecter shutdown clearing up homeless session... 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.235+0000 7fb7e5e49640 10 client.4399.objecter successfully canceled tick 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb7cc0789f0 msgr2=0x7fb7cc07aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb7cc0789f0 0x7fb7cc07aeb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00b8730 tx=0x7fb7d0008030 comp rx=0 tx=0).stop 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 10 monclient: shutdown 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006f390 msgr2=0x7fb7e0142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006f390 0x7fb7e0142030 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400e8a0 tx=0x7fb7d400ed60 comp rx=0 tx=0).stop 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 shutdown_connections 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fb7cc0789f0 0x7fb7cc07aeb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fb7e0074a60 0x7fb7e0143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e006f390 0x7fb7e0142030 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 --2- 172.21.15.139:0/3775305053 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7e006e2f0 0x7fb7e0141720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 >> 172.21.15.139:0/3775305053 conn(0x7fb7e00b7ac0 msgr2=0x7fb7e013e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 shutdown_connections 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 -- 172.21.15.139:0/3775305053 wait complete. 2024-07-26T23:35:30.239 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.236+0000 7fb7e5e49640 1 librados: shutdown 2024-07-26T23:35:30.249 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-07-26T23:35:29.818301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219120,"kb_used_data":2936,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755600,"statfs":{"total":773094113280,"available":772869734400,"internally_reserved":0,"allocated":3006464,"data_stored":1605166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.082214"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.233345+0000","last_change":"2024-07-26T23:35:22.236663+0000","last_active":"2024-07-26T23:35:22.233345+0000","last_peered":"2024-07-26T23:35:22.233345+0000","last_clean":"2024-07-26T23:35:22.233345+0000","last_became_active":"2024-07-26T23:35:18.556903+0000","last_became_peered":"2024-07-26T23:35:18.556903+0000","last_unstale":"2024-07-26T23:35:22.233345+0000","last_undegraded":"2024-07-26T23:35:22.233345+0000","last_fullsized":"2024-07-26T23:35:22.233345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:27.775888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064553026999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.941067+0000","last_change":"2024-07-26T23:35:22.979978+0000","last_active":"2024-07-26T23:35:22.941067+0000","last_peered":"2024-07-26T23:35:22.941067+0000","last_clean":"2024-07-26T23:35:22.941067+0000","last_became_active":"2024-07-26T23:35:18.587303+0000","last_became_peered":"2024-07-26T23:35:18.587303+0000","last_unstale":"2024-07-26T23:35:22.941067+0000","last_undegraded":"2024-07-26T23:35:22.941067+0000","last_fullsized":"2024-07-26T23:35:22.941067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:51:04.403633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.077704690000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.246462+0000","last_change":"2024-07-26T23:35:22.251048+0000","last_active":"2024-07-26T23:35:22.246462+0000","last_peered":"2024-07-26T23:35:22.246462+0000","last_clean":"2024-07-26T23:35:22.246462+0000","last_became_active":"2024-07-26T23:35:17.925979+0000","last_became_peered":"2024-07-26T23:35:17.925979+0000","last_unstale":"2024-07-26T23:35:22.246462+0000","last_undegraded":"2024-07-26T23:35:22.246462+0000","last_fullsized":"2024-07-26T23:35:22.246462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:20.031998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073846415999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.343325+0000","last_change":"2024-07-26T23:35:22.376891+0000","last_active":"2024-07-26T23:35:22.343325+0000","last_peered":"2024-07-26T23:35:22.343325+0000","last_clean":"2024-07-26T23:35:22.343325+0000","last_became_active":"2024-07-26T23:35:18.507138+0000","last_became_peered":"2024-07-26T23:35:18.507138+0000","last_unstale":"2024-07-26T23:35:22.343325+0000","last_undegraded":"2024-07-26T23:35:22.343325+0000","last_fullsized":"2024-07-26T23:35:22.343325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:56:22.382897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047472209000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.220704+0000","last_change":"2024-07-26T23:35:22.220704+0000","last_active":"2024-07-26T23:35:22.220704+0000","last_peered":"2024-07-26T23:35:22.220704+0000","last_clean":"2024-07-26T23:35:22.220704+0000","last_became_active":"2024-07-26T23:35:18.492082+0000","last_became_peered":"2024-07-26T23:35:18.492082+0000","last_unstale":"2024-07-26T23:35:22.220704+0000","last_undegraded":"2024-07-26T23:35:22.220704+0000","last_fullsized":"2024-07-26T23:35:22.220704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:00.865495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098681633000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.055533+0000","last_change":"2024-07-26T23:35:22.093863+0000","last_active":"2024-07-26T23:35:22.055533+0000","last_peered":"2024-07-26T23:35:22.055533+0000","last_clean":"2024-07-26T23:35:22.055533+0000","last_became_active":"2024-07-26T23:35:17.814359+0000","last_became_peered":"2024-07-26T23:35:17.814359+0000","last_unstale":"2024-07-26T23:35:22.055533+0000","last_undegraded":"2024-07-26T23:35:22.055533+0000","last_fullsized":"2024-07-26T23:35:22.055533+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:27:36.663020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060284956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.156536+0000","last_change":"2024-07-26T23:35:22.212052+0000","last_active":"2024-07-26T23:35:22.156536+0000","last_peered":"2024-07-26T23:35:22.156536+0000","last_clean":"2024-07-26T23:35:22.156536+0000","last_became_active":"2024-07-26T23:35:17.846505+0000","last_became_peered":"2024-07-26T23:35:17.846505+0000","last_unstale":"2024-07-26T23:35:22.156536+0000","last_undegraded":"2024-07-26T23:35:22.156536+0000","last_fullsized":"2024-07-26T23:35:22.156536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:31.369623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11625252899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:22.230355+0000","last_change":"2024-07-26T23:35:22.241392+0000","last_active":"2024-07-26T23:35:22.230355+0000","last_peered":"2024-07-26T23:35:22.230355+0000","last_clean":"2024-07-26T23:35:22.230355+0000","last_became_active":"2024-07-26T23:35:19.087174+0000","last_became_peered":"2024-07-26T23:35:19.087174+0000","last_unstale":"2024-07-26T23:35:22.230355+0000","last_undegraded":"2024-07-26T23:35:22.230355+0000","last_fullsized":"2024-07-26T23:35:22.230355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:54:22.975250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.074002362000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'32","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-07-26T23:35:23.723611+0000","last_change":"2024-07-26T23:35:17.404909+0000","last_active":"2024-07-26T23:35:23.723611+0000","last_peered":"2024-07-26T23:35:23.723611+0000","last_clean":"2024-07-26T23:35:23.723611+0000","last_became_active":"2024-07-26T23:35:16.707863+0000","last_became_peered":"2024-07-26T23:35:16.707863+0000","last_unstale":"2024-07-26T23:35:23.723611+0000","last_undegraded":"2024-07-26T23:35:23.723611+0000","last_fullsized":"2024-07-26T23:35:23.723611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_clean_scrub_stamp":"2024-07-26T23:35:01.337925+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:50:39.235610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":20,"seq":85899345932,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":101200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":499021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":101181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":101200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":675840,"data_stored":499021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T23:35:30.250 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T23:35:30.250 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T23:35:30.250 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-07-26T23:35:30.251 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-07-26T23:35:30.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.347+0000 7f82f7b15640 10 monclient: get_monmap_and_config 2024-07-26T23:35:30.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.347+0000 7f82f7b15640 10 monclient: build_initial_monmap 2024-07-26T23:35:30.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.347+0000 7f82f7b15640 10 monclient: monmap: 2024-07-26T23:35:30.349 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:30.349 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:35:30.348692+0000 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:35:30.348692+0000 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:35:30.350 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:30.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 1 Processor -- start 2024-07-26T23:35:30.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 1 -- start start 2024-07-26T23:35:30.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 10 monclient: init 2024-07-26T23:35:30.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:30.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-07-26T23:35:30.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82f00b20e0 addr v1:172.21.15.177:6789/0 2024-07-26T23:35:30.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f82f0144be0 0x7f82f0145010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:30.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.348+0000 7f82f7b15640 10 monclient(hunting): picked mon.noname-c con 0x7f82f0144be0 addr v2:172.21.15.177:3300/0 2024-07-26T23:35:30.353 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f0145550 0x7f82f014bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 10 monclient(hunting): picked mon.noname-b con 0x7f82f0145550 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f0073fd0 con 0x7f82f00b20e0 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f82f00b24c0 con 0x7f82f0144be0 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f82f0074240 con 0x7f82f0145550 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f7b15640 10 monclient(hunting): authenticate will time out at 1363.385051s 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f60b2640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f0145550 0x7f82f014bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:30.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.349+0000 7f82f50b0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f82f0144be0 0x7f82f0145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f0145550 0x7f82f014bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:37914/0 (socket says 172.21.15.139:37914) 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 1 -- 172.21.15.139:0/3606567404 learned_addr learned my addr 172.21.15.139:0/3606567404 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 10 monclient(hunting): get_auth_request con 0x7f82f0145550 auth_method 0 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f50b0640 10 monclient(hunting): get_auth_request con 0x7f82f0144be0 auth_method 0 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f50b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:30.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f50b0640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:30.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f50b0640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:30.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:30.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:30.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f60b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:30.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f50b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:30.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f50b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:30.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.350+0000 7f82f50b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:30.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2024-07-26T23:35:30.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 1 -- 172.21.15.139:0/3606567404 >> v2:172.21.15.177:3300/0 conn(0x7f82f0144be0 msgr2=0x7f82f0145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:35:30.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 1 --2- 172.21.15.139:0/3606567404 >> v2:172.21.15.177:3300/0 conn(0x7f82f0144be0 0x7f82f0145010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 1 -- 172.21.15.139:0/3606567404 >> v1:172.21.15.177:6789/0 conn(0x7f82f00b20e0 legacy=0x7f82f0072b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 10 monclient: _finish_hunting 0 2024-07-26T23:35:30.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 1 monclient: found mon.noname-b 2024-07-26T23:35:30.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:30.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:35:30.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.352+0000 7f82f60b2640 1 -- 172.21.15.139:0/3606567404 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f0072620 con 0x7f82f0145550 2024-07-26T23:35:30.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.354+0000 7f82f60b2640 1 --2- 172.21.15.139:0/3606567404 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f0145550 0x7f82f014bf70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009e50 tx=0x7f82e802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=666614c3d624c2f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:30.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 1 -- 172.21.15.139:0/3606567404 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e802cb00 con 0x7f82f0145550 2024-07-26T23:35:30.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:30.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:30.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: dump: 2024-07-26T23:35:30.359 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:30.360 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: _finish_auth 0 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: _check_auth_tickets 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 1 -- 172.21.15.139:0/3606567404 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e802cca0 con 0x7f82f0145550 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 1 -- 172.21.15.139:0/3606567404 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8034610 con 0x7f82f0145550 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f7b15640 5 monclient: authenticate success, global_id 4349 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f7b15640 10 monclient: get_monmap_and_config success 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f7b15640 4 set_mon_vals no callback set 2024-07-26T23:35:30.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.357+0000 7f82f48af640 10 monclient: dump: 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:30.362 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:30.363 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:30.363 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:30.363 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:30.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 10 monclient: shutdown 2024-07-26T23:35:30.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:30.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 -- 172.21.15.139:0/3606567404 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f0145550 msgr2=0x7f82f014bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 --2- 172.21.15.139:0/3606567404 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f0145550 0x7f82f014bf70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009e50 tx=0x7f82e802bf60 comp rx=0 tx=0).stop 2024-07-26T23:35:30.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 -- 172.21.15.139:0/3606567404 shutdown_connections 2024-07-26T23:35:30.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 --2- 172.21.15.139:0/3606567404 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f0145550 0x7f82f014bf70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 --2- 172.21.15.139:0/3606567404 >> v2:172.21.15.177:3300/0 conn(0x7f82f0144be0 0x7f82f0145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 -- 172.21.15.139:0/3606567404 >> 172.21.15.139:0/3606567404 conn(0x7f82f00b2f60 msgr2=0x7f82f00b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 -- 172.21.15.139:0/3606567404 shutdown_connections 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.358+0000 7f82f7b15640 1 -- 172.21.15.139:0/3606567404 wait complete. 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.359+0000 7f82f7b15640 10 monclient: build_initial_monmap 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.359+0000 7f82f7b15640 10 monclient: monmap: 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:35:30.368 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.359+0000 7f82f7b15640 1 librados: starting msgr at 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.359+0000 7f82f7b15640 1 librados: starting objecter 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.359+0000 7f82f7b15640 1 Processor -- start 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 -- start start 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 librados: setting wanted keys 2024-07-26T23:35:30.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 librados: calling monclient init 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient: init 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient: _reopen_session rank -1 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f00b20e0 0x7f82f0141e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): picked mon.noname-c con 0x7f82f00b20e0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0144be0 0x7f82f0137590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): picked mon.noname-a con 0x7f82f0144be0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82f0145550 0x7f82f0137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): picked mon.noname-b con 0x7f82f0145550 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f82f00722b0 con 0x7f82f0144be0 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f82f013f1a0 con 0x7f82f0145550 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): start opening mon connection 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f82f0145980 con 0x7f82f00b20e0 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): _renew_subs 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.360+0000 7f82f7b15640 10 monclient(hunting): authenticate will time out at 1363.396569s 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f00b20e0 0x7f82f0141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f50b0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0144be0 0x7f82f0137590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f60b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82f0145550 0x7f82f0137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f00b20e0 0x7f82f0141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:37920/0 (socket says 172.21.15.139:37920) 2024-07-26T23:35:30.371 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 1 -- 172.21.15.139:0/108895905 learned_addr learned my addr 172.21.15.139:0/108895905 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 10 monclient(hunting): get_auth_request con 0x7f82f00b20e0 auth_method 0 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f50b0640 10 monclient(hunting): get_auth_request con 0x7f82f0144be0 auth_method 0 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f50b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f50b0640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f50b0640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f60b2640 10 monclient(hunting): get_auth_request con 0x7f82f0145550 auth_method 0 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f60b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f60b2640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:35:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f60b2640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.361+0000 7f82f58b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.362+0000 7f82f60b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.362+0000 7f82f60b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.362+0000 7f82f60b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.362+0000 7f82f50b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.362+0000 7f82f50b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.362+0000 7f82f50b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 1 -- 172.21.15.139:0/108895905 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f00b20e0 msgr2=0x7f82f0141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f00b20e0 0x7f82f0141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 1 -- 172.21.15.139:0/108895905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0144be0 msgr2=0x7f82f0137590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0144be0 0x7f82f0137590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 10 monclient: _finish_hunting 0 2024-07-26T23:35:30.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 1 monclient: found mon.noname-b 2024-07-26T23:35:30.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:35:30.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:35:30.374 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.364+0000 7f82f60b2640 1 -- 172.21.15.139:0/108895905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f0139750 con 0x7f82f0145550 2024-07-26T23:35:30.375 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.366+0000 7f82f60b2640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82f0145550 0x7f82f0137ad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82e8032000 tx=0x7f82e8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.366+0000 7f82e67fc640 10 client.?.objecter ms_handle_connect 0x7f82f0145550 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.366+0000 7f82e67fc640 10 client.?.objecter resend_mon_ops 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 1 -- 172.21.15.139:0/108895905 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8043020 con 0x7f82f0145550 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: dump: 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:30.377 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: _finish_auth 0 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: _check_auth_tickets 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 1 -- 172.21.15.139:0/108895905 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e802ce50 con 0x7f82f0145550 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 5 monclient: authenticate success, global_id 4408 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 10 monclient: _renew_subs 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f0139a30 con 0x7f82f0145550 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 1 -- 172.21.15.139:0/108895905 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8009510 con 0x7f82f0145550 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:35:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e67fc640 10 monclient: dump: 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82e7fff640 4 set_mon_vals no callback set 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:35:30.379 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 10 monclient: _renew_subs 2024-07-26T23:35:30.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:30.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f0138010 con 0x7f82f0145550 2024-07-26T23:35:30.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.370+0000 7f82f7b15640 1 librados: init done 2024-07-26T23:35:30.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.371+0000 7f82f7b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:30.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.371+0000 7f82f7b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:30.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.371+0000 7f82f7b15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:30.380 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.371+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f0145010 con 0x7f82f0145550 2024-07-26T23:35:30.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.392+0000 7f82e67fc640 1 -- 172.21.15.139:0/108895905 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f82e8046020 con 0x7f82f0145550 2024-07-26T23:35:30.393 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.392+0000 7f82e67fc640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f82d4078a20 0x7f82d407aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:35:30.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.392+0000 7f82f58b1640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f82d4078a20 0x7f82d407aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:35:30.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.392+0000 7f82f58b1640 10 monclient: get_auth_request con 0x7f82d4078a20 auth_method 0 2024-07-26T23:35:30.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.393+0000 7f82f58b1640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f82d4078a20 0x7f82d407aee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0018a0 tx=0x7f82ec009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:35:30.394 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.393+0000 7f82e67fc640 10 client.4408.objecter ms_handle_connect 0x7f82d4078a20 2024-07-26T23:35:30.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.395+0000 7f82e67fc640 1 -- 172.21.15.139:0/108895905 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5266+0+0 (secure 0 0 0) 0x7f82e807ff60 con 0x7f82f0145550 2024-07-26T23:35:30.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.395+0000 7f82e67fc640 10 client.4408.objecter ms_dispatch 0x7f82f0000b90 osd_map(28..28 src has 1..28) 2024-07-26T23:35:30.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.395+0000 7f82e67fc640 3 client.4408.objecter handle_osd_map got epochs [28,28] > 0 2024-07-26T23:35:30.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.395+0000 7f82e67fc640 3 client.4408.objecter handle_osd_map decoding full epoch 28 2024-07-26T23:35:30.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.395+0000 7f82e67fc640 20 client.4408.objecter dump_active .. 0 homeless 2024-07-26T23:35:30.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.631+0000 7f82e67fc640 1 -- 172.21.15.139:0/108895905 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f82e804d030 con 0x7f82f0145550 2024-07-26T23:35:30.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.631+0000 7f82e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:35:30.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.631+0000 7f82e67fc640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:35:30.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.884+0000 7f82f7b15640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-07-26T23:35:30.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.884+0000 7f82f7b15640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-07-26T23:35:30.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.884+0000 7f82f7b15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:35:30.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.884+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f82f006b560 con 0x7f82f0145550 2024-07-26T23:35:30.892 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.891+0000 7f82e67fc640 1 -- 172.21.15.139:0/108895905 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f82e8083cd0 con 0x7f82f0145550 2024-07-26T23:35:30.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.891+0000 7f82e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-07-26T23:35:30.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.891+0000 7f82e67fc640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:35:30.893 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:35:30.893 INFO:teuthology.orchestra.run.smithi139.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T23:35:30.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 10 librados: watch_flush enter 2024-07-26T23:35:30.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 10 librados: watch_flush exit 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 20 client.4408.objecter shutdown clearing up homeless session... 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 10 client.4408.objecter successfully canceled tick 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f82d4078a20 msgr2=0x7f82d407aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f82d4078a20 0x7f82d407aee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0018a0 tx=0x7f82ec009040 comp rx=0 tx=0).stop 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 10 monclient: shutdown 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82f0145550 msgr2=0x7f82f0137ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82f0145550 0x7f82f0137ad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82e8032000 tx=0x7f82e8004a40 comp rx=0 tx=0).stop 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 shutdown_connections 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f82d4078a20 0x7f82d407aee0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82f0145550 0x7f82f0137ad0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f82f0144be0 0x7f82f0137590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 --2- 172.21.15.139:0/108895905 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f82f00b20e0 0x7f82f0141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:35:30.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.893+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 >> 172.21.15.139:0/108895905 conn(0x7f82f00b2f60 msgr2=0x7f82f006c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:35:30.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.894+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 shutdown_connections 2024-07-26T23:35:30.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.894+0000 7f82f7b15640 1 -- 172.21.15.139:0/108895905 wait complete. 2024-07-26T23:35:30.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:35:30.894+0000 7f82f7b15640 1 librados: shutdown 2024-07-26T23:35:30.906 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-07-26T23:35:30.906 INFO:teuthology.run_tasks:Running task workunit... 2024-07-26T23:35:31.636 INFO:tasks.workunit:Pulling workunits from ref 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b 2024-07-26T23:35:31.637 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-26T23:35:31.637 DEBUG:teuthology.orchestra.run.smithi139:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-26T23:35:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T23:35:31.654 INFO:teuthology.orchestra.run.smithi139.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-07-26T23:35:31.654 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-07-26T23:35:31.710 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-07-26T23:35:31.710 DEBUG:teuthology.orchestra.run.smithi139:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-07-26T23:35:31.769 INFO:tasks.workunit:timeout=6h 2024-07-26T23:35:31.769 INFO:tasks.workunit:cleanup=True 2024-07-26T23:35:31.769 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 67f28c2a04ad1fe90ea612a39329cc6af2a92d8b 2024-07-26T23:35:31.825 INFO:tasks.workunit.client.0.smithi139.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-26T23:35:32.076 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:39.747 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.082 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:39.754 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.133 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:39.796 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.135 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:39.798 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.139 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:39.811 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.143 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:39.814 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.148 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:39.819 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.177 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:39.840 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:32.188 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:39.860 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.107 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:59.778 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.182 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:59.854 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.203 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:01:59.874 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.232 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:59.895 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.237 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:59.899 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.251 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:01:59.914 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.274 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:59.945 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.276 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:59.947 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:35:52.299 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:01:59.970 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.263 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:02:19.934 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.267 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:02:19.939 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.289 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:02:19.960 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.325 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:19.988 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.327 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:19.990 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.335 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:02:20.006 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.340 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:02:20.012 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.343 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:20.006 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:12.366 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:02:20.037 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:20.675 INFO:tasks.workunit.client.0.smithi139.stderr:Note: switching to '67f28c2a04ad1fe90ea612a39329cc6af2a92d8b'. 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-07-26T23:36:20.676 INFO:tasks.workunit.client.0.smithi139.stderr: git switch -c 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr:Or undo this operation with: 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr: git switch - 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-07-26T23:36:20.677 INFO:tasks.workunit.client.0.smithi139.stderr:HEAD is now at 67f28c2a04a Merge branch 'ceph-nvmeof-mon' of https://github.com/baum/ceph into wip-yuri-testing-2024-07-26-0628 2024-07-26T23:36:20.680 DEBUG:teuthology.orchestra.run.smithi139:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-26T23:36:20.757 INFO:tasks.workunit.client.0.smithi139.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-26T23:36:20.759 INFO:tasks.workunit.client.0.smithi139.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T23:36:20.759 INFO:tasks.workunit.client.0.smithi139.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-26T23:36:21.325 INFO:tasks.workunit.client.0.smithi139.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-26T23:36:21.378 INFO:tasks.workunit.client.0.smithi139.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-26T23:36:21.417 INFO:tasks.workunit.client.0.smithi139.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T23:36:21.418 INFO:tasks.workunit.client.0.smithi139.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T23:36:21.418 INFO:tasks.workunit.client.0.smithi139.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-26T23:36:21.457 INFO:tasks.workunit.client.0.smithi139.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T23:36:21.460 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-07-26T23:36:21.461 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-26T23:36:21.516 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-07-26T23:36:21.518 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-07-26T23:36:21.518 DEBUG:teuthology.orchestra.run.smithi139:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=67f28c2a04ad1fe90ea612a39329cc6af2a92d8b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-07-26T23:36:21.577 INFO:tasks.workunit.client.0.smithi139.stderr:+ parallel=1 2024-07-26T23:36:21.577 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' '' = --serial ']' 2024-07-26T23:36:21.577 INFO:tasks.workunit.client.0.smithi139.stderr:+ crimson=0 2024-07-26T23:36:21.578 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' '' = --crimson ']' 2024-07-26T23:36:21.578 INFO:tasks.workunit.client.0.smithi139.stderr:+ color= 2024-07-26T23:36:21.578 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -t 1 ']' 2024-07-26T23:36:21.578 INFO:tasks.workunit.client.0.smithi139.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-07-26T23:36:21.578 INFO:tasks.workunit.client.0.smithi139.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-07-26T23:36:21.578 INFO:tasks.workunit.client.0.smithi139.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-07-26T23:36:21.580 INFO:tasks.workunit.client.0.smithi139.stderr:+ declare -A pids 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_aio 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_aio' 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_aio 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_aio 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40444 2024-07-26T23:36:21.581 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_aio on pid 40444' 2024-07-26T23:36:21.582 INFO:tasks.workunit.client.0.smithi139.stdout:test api_aio on pid 40444 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40444 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_aio_pp 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_aio_pp' 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_aio_pp 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.585 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.586 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.587 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.588 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.588 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_aio_pp 2024-07-26T23:36:21.588 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40451 2024-07-26T23:36:21.588 INFO:tasks.workunit.client.0.smithi139.stdout:test api_aio_pp on pid 40451 2024-07-26T23:36:21.593 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_aio_pp on pid 40451' 2024-07-26T23:36:21.593 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40451 2024-07-26T23:36:21.593 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.593 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.593 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_io 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_io' 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_io 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.595 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.596 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.597 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.598 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.598 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.598 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.598 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.598 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.599 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.599 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_io 2024-07-26T23:36:21.599 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40460 2024-07-26T23:36:21.599 INFO:tasks.workunit.client.0.smithi139.stdout:test api_io on pid 40460 2024-07-26T23:36:21.606 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_io on pid 40460' 2024-07-26T23:36:21.606 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40460 2024-07-26T23:36:21.606 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.606 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_io_pp 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_io_pp' 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_io_pp 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.607 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.609 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.610 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_io_pp 2024-07-26T23:36:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40471 2024-07-26T23:36:21.614 INFO:tasks.workunit.client.0.smithi139.stdout:test api_io_pp on pid 40471 2024-07-26T23:36:21.621 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_io_pp on pid 40471' 2024-07-26T23:36:21.621 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40471 2024-07-26T23:36:21.621 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.621 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.621 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_asio 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_asio' 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_asio 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.622 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.623 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.624 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.624 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.624 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.624 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.624 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.624 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.624 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.625 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.626 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.627 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.628 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.629 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.630 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.630 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_asio 2024-07-26T23:36:21.630 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.630 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40487 2024-07-26T23:36:21.630 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_asio on pid 40487' 2024-07-26T23:36:21.631 INFO:tasks.workunit.client.0.smithi139.stdout:test api_asio on pid 40487 2024-07-26T23:36:21.638 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40487 2024-07-26T23:36:21.638 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.638 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.638 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-07-26T23:36:21.638 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_list 2024-07-26T23:36:21.638 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_list' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.639 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_list 2024-07-26T23:36:21.640 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.641 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.642 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.643 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.644 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.645 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40444/exe 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_list 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40504 2024-07-26T23:36:21.646 INFO:tasks.workunit.client.0.smithi139.stdout:test api_list on pid 40504 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_list on pid 40504' 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40504 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.656 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_lock 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.657 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_lock' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.658 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_lock 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.659 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.660 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_aio.log 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.661 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.663 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_aio: /' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.664 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.665 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40451/exe 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.666 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_lock 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40525 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.667 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.668 INFO:tasks.workunit.client.0.smithi139.stdout:test api_lock on pid 40525 2024-07-26T23:36:21.676 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_lock on pid 40525' 2024-07-26T23:36:21.676 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40525 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.677 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_lock_pp 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.678 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_lock_pp' 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.679 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.680 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.681 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.682 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_aio_pp: /' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40460/exe 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_lock_pp 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_lock_pp 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40542 2024-07-26T23:36:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_lock_pp on pid 40542' 2024-07-26T23:36:21.685 INFO:tasks.workunit.client.0.smithi139.stdout:test api_lock_pp on pid 40542 2024-07-26T23:36:21.691 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40542 2024-07-26T23:36:21.691 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.691 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.691 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.691 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_misc 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.692 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_misc' 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40471/exe 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.693 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_misc 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.694 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.695 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_io.log 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_io: /' 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.696 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_misc 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.697 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40558 2024-07-26T23:36:21.698 INFO:tasks.workunit.client.0.smithi139.stdout:test api_misc on pid 40558 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_misc on pid 40558' 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40558 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.705 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.706 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_misc_pp 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_misc_pp' 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.707 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_io_pp: /' 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.708 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.709 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_misc_pp 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.710 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.711 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.712 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.713 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.713 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.713 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.713 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_misc_pp 2024-07-26T23:36:21.713 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40580 2024-07-26T23:36:21.713 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_misc_pp on pid 40580' 2024-07-26T23:36:21.713 INFO:tasks.workunit.client.0.smithi139.stdout:test api_misc_pp on pid 40580 2024-07-26T23:36:21.724 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40580 2024-07-26T23:36:21.724 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.724 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.724 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.724 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.725 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.726 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.727 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.728 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_tier_pp 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_tier_pp' 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.729 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.730 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_tier_pp 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.732 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40487/exe 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.733 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.734 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.735 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_tier_pp 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40602 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_tier_pp on pid 40602' 2024-07-26T23:36:21.736 INFO:tasks.workunit.client.0.smithi139.stdout:test api_tier_pp on pid 40602 2024-07-26T23:36:21.746 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40602 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_pool 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_pool' 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.747 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_asio.log 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.748 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.749 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.750 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.751 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.752 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_pool 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_asio: /' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.753 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.754 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.754 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.754 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.754 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.754 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.754 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.755 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40504/exe 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.757 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.757 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.757 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_pool 2024-07-26T23:36:21.757 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.757 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.757 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.757 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40623 2024-07-26T23:36:21.758 INFO:tasks.workunit.client.0.smithi139.stdout:test api_pool on pid 40623 2024-07-26T23:36:21.774 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_pool on pid 40623' 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40623 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.775 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_snapshots 2024-07-26T23:36:21.776 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_snapshots' 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40525/exe 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.777 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.778 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_list.log 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.779 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.780 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_list: /' 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.781 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.782 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.783 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_lock: /' 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_lock.log 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.784 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.785 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.786 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.787 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40542/exe 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.788 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_lock_pp: /' 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40558/exe 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_snapshots 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_snapshots 2024-07-26T23:36:21.789 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.790 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40663 2024-07-26T23:36:21.790 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_snapshots on pid 40663' 2024-07-26T23:36:21.790 INFO:tasks.workunit.client.0.smithi139.stdout:test api_snapshots on pid 40663 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40663 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.811 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_misc: /' 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.812 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.813 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.814 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.815 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.816 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.817 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.818 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40580/exe 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.819 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_snapshots_pp 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_snapshots_pp' 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_misc.log 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.820 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_misc_pp: /' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.821 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.822 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_snapshots_pp 2024-07-26T23:36:21.823 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.827 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_snapshots_pp 2024-07-26T23:36:21.827 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40725 2024-07-26T23:36:21.827 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_snapshots_pp on pid 40725' 2024-07-26T23:36:21.827 INFO:tasks.workunit.client.0.smithi139.stdout:test api_snapshots_pp on pid 40725 2024-07-26T23:36:21.843 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40725 2024-07-26T23:36:21.843 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.843 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.843 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-07-26T23:36:21.843 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.843 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.844 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_stat 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.845 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_stat' 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.846 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.847 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40623/exe 2024-07-26T23:36:21.848 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.849 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.850 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40663/exe 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.851 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.852 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_pool.log 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_stat 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.853 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.854 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_pool: /' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.855 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_snapshots: /' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.856 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40602/exe 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_stat 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40794 2024-07-26T23:36:21.857 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_stat on pid 40794' 2024-07-26T23:36:21.858 INFO:tasks.workunit.client.0.smithi139.stdout:test api_stat on pid 40794 2024-07-26T23:36:21.859 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40794 2024-07-26T23:36:21.859 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.859 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.859 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-07-26T23:36:21.859 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_stat_pp 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_stat_pp' 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.860 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [----------] Global test environment set-up. 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [----------] 12 tests from AsioRados 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.709+0000 7f6e38a3f780 10 monclient: get_monmap_and_config 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.709+0000 7f6e38a3f780 10 monclient: build_initial_monmap 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.709+0000 7f6e38a3f780 10 monclient: monmap: 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: epoch 0 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: last_changed 0.000000 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: created 0.000000 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: min_mon_release 0 (unknown) 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: election_strategy: 1 2024-07-26T23:36:21.861 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.709+0000 7f6e38a3f780 1 Processor -- start 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.709+0000 7f6e38a3f780 1 -- start start 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.709+0000 7f6e38a3f780 10 monclient: init 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient: _reopen_session rank -1 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d228360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): picked mon.noname-c con 0x55d38d1eb400 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:36:21.862 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d38d22ca40 0x55d38d22a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): picked mon.noname-a con 0x55d38d22ca40 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d2bc700 0x55d38d2c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): picked mon.noname-b con 0x55d38d2bc700 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): start opening mon connection 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d38d146a30 con 0x55d38d22ca40 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): start opening mon connection 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d38d1a4960 con 0x55d38d2bc700 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): start opening mon connection 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x55d38d226cc0 con 0x55d38d1eb400 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): _renew_subs 2024-07-26T23:36:21.863 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.710+0000 7f6e38a3f780 10 monclient(hunting): authenticate will time out at 1414.746867s 2024-07-26T23:36:21.867 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.867 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.867 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.868 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.869 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_stat_pp 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.871 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.715+0000 7f6e36534640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d228360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.716+0000 7f6e36534640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d228360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:43082/0 (socket says 172.21.15.139:43082) 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.716+0000 7f6e36534640 1 -- 172.21.15.139:0/4151594553 learned_addr learned my addr 172.21.15.139:0/4151594553 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.716+0000 7f6e36534640 10 monclient(hunting): get_auth_request con 0x55d38d1eb400 auth_method 0 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.716+0000 7f6e36534640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.716+0000 7f6e36534640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.716+0000 7f6e36534640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.718+0000 7f6e36534640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.718+0000 7f6e36534640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.718+0000 7f6e36534640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.719+0000 7f6e36d35640 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d2bc700 0x55d38d2c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:36:21.872 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.719+0000 7f6e36d35640 10 monclient(hunting): get_auth_request con 0x55d38d2bc700 auth_method 0 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.719+0000 7f6e36d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.719+0000 7f6e36d35640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.719+0000 7f6e36d35640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 1 -- 172.21.15.139:0/4151594553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d2bc700 msgr2=0x55d38d2c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d2bc700 0x55d38d2c3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 1 -- 172.21.15.139:0/4151594553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d38d22ca40 msgr2=0x55d38d22a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d38d22ca40 0x55d38d22a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 10 monclient: _finish_hunting 0 2024-07-26T23:36:21.873 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 1 monclient: found mon.noname-c 2024-07-26T23:36:21.877 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.877 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.877 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.877 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.877 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_tier_pp: /' 2024-07-26T23:36:21.877 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.877 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.878 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.879 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_stat_pp 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40831 2024-07-26T23:36:21.880 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_stat_pp on pid 40831' 2024-07-26T23:36:21.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 20 monclient: _test api_stat_pp on pid 40831 2024-07-26T23:36:21.887 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40831 2024-07-26T23:36:21.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.946 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.947 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40725/exe 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.949 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_watch_notify 2024-07-26T23:36:21.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_watch_notify' 2024-07-26T23:36:21.951 INFO:tasks.workunit.client.0.smithi139.stdout:un_backoff reopen_interval_multipler now 1 2024-07-26T23:36:21.951 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:36:21.951 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.723+0000 7f6e36534640 1 -- 172.21.15.139:0/4151594553 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d38d120080 con 0x55d38d1eb400 2024-07-26T23:36:21.951 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.728+0000 7f6e36534640 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d228360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00a590 tx=0x7f6e2c00aa50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c671a9fb86e5382 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:36:21.951 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.748+0000 7f6e34d31640 1 -- 172.21.15.139:0/4151594553 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c00b980 con 0x55d38d1eb400 2024-07-26T23:36:21.951 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:36:21.951 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: dump: 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: epoch 1 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: min_mon_release 19 (squid) 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: election_strategy: 1 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: _finish_auth 0 2024-07-26T23:36:21.952 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: _check_auth_tickets 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 1 -- 172.21.15.139:0/4151594553 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e2c00f070 con 0x55d38d1eb400 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: handle_config config(0 keys) 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 1 -- 172.21.15.139:0/4151594553 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c00bd80 con 0x55d38d1eb400 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e34d31640 10 monclient: dump: 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: epoch 1 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: min_mon_release 19 (squid) 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: election_strategy: 1 2024-07-26T23:36:21.953 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:36:21.954 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:36:21.954 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:36:21.954 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.954 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e38a3f780 5 monclient: authenticate success, global_id 4370 2024-07-26T23:36:21.954 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e38a3f780 10 monclient: get_monmap_and_config success 2024-07-26T23:36:21.955 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.955 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_watch_notify 2024-07-26T23:36:21.955 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.955 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.955 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.956 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.956 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.956 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.956 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.956 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.956 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.749+0000 7f6e38a3f780 4 set_mon_vals no callback set 2024-07-26T23:36:21.956 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 10 monclient: shutdown 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 -- 172.21.15.139:0/4151594553 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 msgr2=0x55d38d228360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d228360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00a590 tx=0x7f6e2c00aa50 comp rx=0 tx=0).stop 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 -- 172.21.15.139:0/4151594553 shutdown_connections 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d2bc700 0x55d38d2c3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d38d22ca40 0x55d38d22a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/4151594553 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d228360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 -- 172.21.15.139:0/4151594553 >> 172.21.15.139:0/4151594553 conn(0x55d38d230760 msgr2=0x55d38d22f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 -- 172.21.15.139:0/4151594553 shutdown_connections 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 1 -- 172.21.15.139:0/4151594553 wait complete. 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 10 monclient: build_initial_monmap 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.750+0000 7f6e38a3f780 10 monclient: monmap: 2024-07-26T23:36:21.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: epoch 0 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: last_changed 0.000000 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: created 0.000000 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: min_mon_release 0 (unknown) 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: election_strategy: 1 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.751+0000 7f6e38a3f780 1 librados: starting msgr at 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.751+0000 7f6e38a3f780 1 librados: starting objecter 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.751+0000 7f6e38a3f780 1 Processor -- start 2024-07-26T23:36:21.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.751+0000 7f6e38a3f780 1 -- start start 2024-07-26T23:36:21.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.751+0000 7f6e38a3f780 1 librados: setting wanted keys 2024-07-26T23:36:21.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.751+0000 7f6e38a3f780 1 librados: calling monclient init 2024-07-26T23:36:21.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.751+0000 7f6e38a3f780 10 monclient: init 2024-07-26T23:36:21.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient: _reopen_session rank -1 2024-07-26T23:36:21.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:36:21.960 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.960 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:21.960 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:21.960 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_watch_notify 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40884 2024-07-26T23:36:21.961 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d2b0test api_watch_notify on pid 40884 2024-07-26T23:36:21.972 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_watch_notify on pid 40884' 2024-07-26T23:36:21.972 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40884 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_watch_notify_pp 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_watch_notify_pp' 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.973 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.974 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.975 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:21.976 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_watch_notify_pp 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:21.978 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.979 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:21.980 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.981 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_watch_notify_pp 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40922 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.982 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.983 INFO:tasks.workunit.client.0.smithi139.stdout:test api_watch_notify_pp on pid 40922 2024-07-26T23:36:21.988 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:21.988 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_watch_notify_pp on pid 40922' 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40922 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_cmd 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_cmd' 2024-07-26T23:36:21.989 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40794/exe 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_cmd 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40831/exe 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:21.990 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.991 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.992 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.992 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.992 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:21.992 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:21.993 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.993 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.994 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:21.995 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-07-26T23:36:21.996 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_stat: /' 2024-07-26T23:36:21.996 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:21.996 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-07-26T23:36:21.996 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_cmd 2024-07-26T23:36:21.996 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40940 2024-07-26T23:36:21.996 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_cmd on pid 40940' 2024-07-26T23:36:21.996 INFO:tasks.workunit.client.0.smithi139.stdout:test api_cmd on pid 40940 2024-07-26T23:36:21.997 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40940 2024-07-26T23:36:21.997 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:21.997 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_stat.log 2024-07-26T23:36:21.997 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout:5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): picked mon.noname-c con 0x55d38d1eb400 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d22ca40 0x55d38d2b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): picked mon.noname-b con 0x55d38d22ca40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d38d2bc700 0x55d38d2b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): picked mon.noname-a con 0x55d38d2bc700 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): start opening mon connection 2024-07-26T23:36:21.998 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x55d38d22df90 con 0x55d38d2bc700 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): start opening mon connection 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d38d1e6340 con 0x55d38d22ca40 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): start opening mon connection 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x55d38d2c4440 con 0x55d38d1eb400 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): _renew_subs 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.752+0000 7f6e38a3f780 10 monclient(hunting): authenticate will time out at 1414.788566s 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.769+0000 7f6e36534640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d2b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.769+0000 7f6e36534640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d2b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:43112/0 (socket says 172.21.15.139:43112) 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.769+0000 7f6e36534640 1 -- 172.21.15.139:0/1617108442 learned_addr learned my addr 172.21.15.139:0/1617108442 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:36:21.999 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.769+0000 7f6e36534640 10 monclient(hunting): get_auth_request con 0x55d38d1eb400 auth_method 0 2024-07-26T23:36:22.000 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.769+0000 7f6e36534640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:36:22.000 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.769+0000 7f6e36534640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:36:22.000 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.769+0000 7f6e36534640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:36:22.000 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.770+0000 7f6e36534640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:36:22.000 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.770+0000 7f6e36534640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:36:22.000 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.770+0000 7f6e36534640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:36:22.006 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_cmd_pp 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_stat_pp: /' 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_cmd_pp' 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.007 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_cmd_pp 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.008 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.009 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.010 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_cmd_pp 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40958 2024-07-26T23:36:22.011 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_cmd_pp on pid 40958' 2024-07-26T23:36:22.012 INFO:tasks.workunit.client.0.smithi139.stdout: test api_cmd_pp on pid 40958 2024-07-26T23:36:22.023 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40958 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_service 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.024 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_service' 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.025 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.026 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.027 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.028 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.029 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40884/exe 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.030 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.031 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_watch_notify: /' 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_service 2024-07-26T23:36:22.033 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_service 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40992 2024-07-26T23:36:22.035 INFO:tasks.workunit.client.0.smithi139.stdout:test api_service on pid 40992 2024-07-26T23:36:22.041 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_service on pid 40992' 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=40992 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_service_pp 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.043 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.044 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_service_pp' 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_service_pp 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.045 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.046 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.047 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_service_pp 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41008 2024-07-26T23:36:22.048 INFO:tasks.workunit.client.0.smithi139.stdout:test api_service_pp on pid 41008 2024-07-26T23:36:22.061 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_service_pp on pid 41008' 2024-07-26T23:36:22.061 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41008 2024-07-26T23:36:22.061 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:22.061 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.061 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.061 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_c_write_operations 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_c_write_operations' 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.062 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.063 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.064 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.065 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.066 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40922/exe 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.067 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_c_write_operations 2024-07-26T23:36:22.068 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.069 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.070 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.071 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.072 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_c_write_operations 2024-07-26T23:36:22.073 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41040 2024-07-26T23:36:22.074 INFO:tasks.workunit.client.0.smithi139.stdout:test api_c_write_operations on pid 41040 2024-07-26T23:36:22.088 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_c_write_operations on pid 41040' 2024-07-26T23:36:22.088 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41040 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40940/exe 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.091 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.092 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.093 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.096 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_cmd.log 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.097 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s api_c_read_operations 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' api_c_read_operations' 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.098 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo api_c_read_operations 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.099 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.100 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.101 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.102 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_cmd: /' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.103 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.104 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.104 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=api_c_read_operations 2024-07-26T23:36:22.104 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41079 2024-07-26T23:36:22.104 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.104 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.104 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test api_c_read_operations on pid 41079' 2024-07-26T23:36:22.104 INFO:tasks.workunit.client.0.smithi139.stdout:test api_c_read_operations on pid 41079 2024-07-26T23:36:22.111 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41079 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.112 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s list_parallel 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40992/exe 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' list_parallel' 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.113 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.114 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.115 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.116 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.116 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.116 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.116 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.116 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.116 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.116 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.117 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_service.log 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_service: /' 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.118 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.119 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo list_parallel 2024-07-26T23:36:22.120 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=list_parallel 2024-07-26T23:36:22.120 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41101 2024-07-26T23:36:22.120 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test list_parallel on pid 41101' 2024-07-26T23:36:22.120 INFO:tasks.workunit.client.0.smithi139.stdout:test list_parallel on pid 41101 2024-07-26T23:36:22.135 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41101 2024-07-26T23:36:22.135 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:22.135 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.135 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/40958/exe 2024-07-26T23:36:22.135 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.135 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.135 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.136 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.137 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.138 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.139 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s open_pools_parallel 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.140 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' open_pools_parallel' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.141 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41008/exe 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.142 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.143 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.144 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.145 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-07-26T23:36:22.146 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_service_pp: /' 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.147 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.148 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo open_pools_parallel 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 1 -- 172.21.15.139:0/1617108442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d22ca40 msgr2=0x55d38d2b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d38d22ca40 0x55d38d2b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 1 -- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d38d2bc700 msgr2=0x55d38d2b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d38d2bc700 0x55d38d2b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:22.149 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 10 monclient: _finish_hunting 0 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 1 monclient: found mon.noname-c 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.775+0000 7f6e36534640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d38d2c2a80 con 0x55d38d1eb400 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.784+0000 7f6e36534640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d2b05a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00aa90 tx=0x7f6e2c002fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.784+0000 7f6e277fe640 10 client.?.objecter ms_handle_connect 0x55d38d1eb400 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.784+0000 7f6e277fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c024070 con 0x55d38d1eb400 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: dump: 2024-07-26T23:36:22.150 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: epoch 1 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: min_mon_release 19 (squid) 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: election_strategy: 1 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: _finish_auth 0 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: _check_auth_tickets 2024-07-26T23:36:22.151 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41040/exe 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.158 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.159 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.160 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=open_pools_parallel 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41151 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test open_pools_parallel on pid 41151' 2024-07-26T23:36:22.161 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 1test open_pools_parallel on pid 41151 2024-07-26T23:36:22.167 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41151 2024-07-26T23:36:22.167 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-07-26T23:36:22.167 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.167 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-07-26T23:36:22.167 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s delete_pools_parallel 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' delete_pools_parallel' 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo delete_pools_parallel 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.168 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.169 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.170 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.171 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.172 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=delete_pools_parallel 2024-07-26T23:36:22.172 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.172 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.172 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.172 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41176 2024-07-26T23:36:22.172 INFO:tasks.workunit.client.0.smithi139.stdout:test delete_pools_parallel on pid 41176 2024-07-26T23:36:22.182 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test delete_pools_parallel on pid 41176' 2024-07-26T23:36:22.182 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41176 2024-07-26T23:36:22.182 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.182 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.182 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.182 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.182 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.183 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.184 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s cls 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' cls' 2024-07-26T23:36:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo cls 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.187 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.188 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=cls 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41204 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test cls on pid 41204' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.190 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.191 INFO:tasks.workunit.client.0.smithi139.stdout:test cls on pid 41204 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41204 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.199 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.200 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.201 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41079/exe 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s cmd 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' cmd' 2024-07-26T23:36:22.202 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo cmd 2024-07-26T23:36:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.207 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.208 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=cmd 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41234 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test cmd on pid 41234' 2024-07-26T23:36:22.209 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.210 INFO:tasks.workunit.client.0.smithi139.stdout:test cmd on pid 41234 2024-07-26T23:36:22.220 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41234 2024-07-26T23:36:22.220 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.220 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.221 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s handler_error 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.222 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' handler_error' 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-07-26T23:36:22.223 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo handler_error 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41101/exe 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.224 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.225 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.226 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.228 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.229 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41176/exe 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.230 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.231 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.232 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.233 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.234 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.235 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.235 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.235 INFO:tasks.workunit.client.0.smithi139.stdout: -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e2c00f050 con 0x55d38d1eb400 2024-07-26T23:36:22.235 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:36:22.235 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c014ca0 con 0x55d38d1eb400 2024-07-26T23:36:22.235 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:36:22.235 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e277fe640 10 monclient: dump: 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: epoch 1 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: min_mon_release 19 (squid) 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: election_strategy: 1 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:22.236 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 5 monclient: authenticate success, global_id 4379 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 10 monclient: _renew_subs 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d38d2c2d60 con 0x55d38d1eb400 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 10 monclient: _renew_subs 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55d38d2c34f0 con 0x55d38d1eb400 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 1 librados: init done 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e38a3f780 10 librados: wait_for_osdmap waiting 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.868+0000 7f6e27fff640 4 set_mon_vals no callback set 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.921+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f6e2c01e3d0 con 0x55d38d1eb400 2024-07-26T23:36:22.237 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.921+0000 7f6e277fe640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6e18076c10 0x7f6e180790d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:22.238 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.925+0000 7f6e35d33640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6e18076c10 0x7f6e180790d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:36:22.238 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.925+0000 7f6e35d33640 10 monclient: get_auth_request con 0x7f6e18076c10 auth_method 0 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ list_parallel: /' 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_list_parallel.log 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=handler_error 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41265 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test handler_error on pid 41265' 2024-07-26T23:36:22.239 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.926+0000 7f6e35d33640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0test handler_error on pid 41265 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41265 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s io 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' io' 2024-07-26T23:36:22.249 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.250 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.251 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.252 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.253 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-07-26T23:36:22.254 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.255 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.256 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.256 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.256 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.256 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.256 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.256 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.256 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo io 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.257 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=io 2024-07-26T23:36:22.258 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41291 2024-07-26T23:36:22.258 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test io on pid 41291' 2024-07-26T23:36:22.258 INFO:tasks.workunit.client.0.smithi139.stdout:test io on pid 41291 2024-07-26T23:36:22.272 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41291 2024-07-26T23:36:22.272 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.272 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s ec_io 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.273 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' ec_io' 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo ec_io 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.275 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.276 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41151/exe 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.277 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.278 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.279 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.280 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.281 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.282 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-07-26T23:36:22.283 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=ec_io 2024-07-26T23:36:22.284 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41328 2024-07-26T23:36:22.284 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test ec_io on pid 41328' 2024-07-26T23:36:22.284 INFO:tasks.workunit.client.0.smithi139.stdout:test ec_io on pid 41328 2024-07-26T23:36:22.285 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41328 2024-07-26T23:36:22.285 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.285 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.285 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s list 2024-07-26T23:36:22.286 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' list' 2024-07-26T23:36:22.286 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo list 2024-07-26T23:36:22.286 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.286 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=list 2024-07-26T23:36:22.286 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41341 2024-07-26T23:36:22.286 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test list on pid 41341' 2024-07-26T23:36:22.286 INFO:tasks.workunit.client.0.smithi139.stdout:test list on pid 41341 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41341 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.299 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41204/exe 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s ec_list 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' ec_list' 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.300 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.301 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.302 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.303 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_cls 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.304 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.305 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.306 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.307 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_cls.log 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.308 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.309 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ cls: /' 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo ec_list 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41234/exe 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=ec_list 2024-07-26T23:36:22.310 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41388 2024-07-26T23:36:22.311 INFO:tasks.workunit.client.0.smithi139.stdout:test ec_list on pid 41388 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test ec_list on pid 41388' 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41388 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.322 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s misc 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' misc' 2024-07-26T23:36:22.323 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.324 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo misc 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.325 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.326 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.327 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ cmd: /' 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_cmd 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_cmd.log 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.328 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.329 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=misc 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41432 2024-07-26T23:36:22.330 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test misc on pid 41432' 2024-07-26T23:36:22.331 INFO:tasks.workunit.client.0.smithi139.stdout:test misc on pid 41432 2024-07-26T23:36:22.341 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41432 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s pool 2024-07-26T23:36:22.342 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' pool' 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.343 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41265/exe 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.344 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo pool 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.345 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.346 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.348 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.349 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_handler_error 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_handler_error.log 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ handler_error: /' 2024-07-26T23:36:22.350 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41341/exe 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=pool 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41469 2024-07-26T23:36:22.351 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test pool on pid 41469' 2024-07-26T23:36:22.352 INFO:tasks.workunit.client.0.smithi139.stdout:test pool on pid 41469 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41469 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.361 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.362 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.363 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.364 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s read_operations 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' read_operations' 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.365 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.366 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ list: /' 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_list.log 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.368 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.369 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo read_operations 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.370 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=read_operations 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41491 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test read_operations on pid 41491' 2024-07-26T23:36:22.371 INFO:tasks.workunit.client.0.smithi139.stdout:test read_operations on pid 41491 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41491 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.387 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_list 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s snapshots 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' snapshots' 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.388 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo snapshots 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.389 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41291/exe 2024-07-26T23:36:22.390 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.391 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.392 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41328/exe 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.393 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.394 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.395 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_ec_io 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.396 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.397 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_ec_io.log 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ ec_io: /' 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.398 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.399 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.400 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_io 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.401 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.402 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.403 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=snapshots 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.404 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41546 2024-07-26T23:36:22.405 INFO:tasks.workunit.client.0.smithi139.stdout:test snapshots on pid 41546 2024-07-26T23:36:22.411 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test snapshots on pid 41546' 2024-07-26T23:36:22.411 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41546 2024-07-26T23:36:22.411 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.411 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.411 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s watch_notify 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' watch_notify' 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_io.log 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41469/exe 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo watch_notify 2024-07-26T23:36:22.414 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.415 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.416 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.417 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.417 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.417 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.417 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.417 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.417 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.417 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.418 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=watch_notify 2024-07-26T23:36:22.419 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.419 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41559 2024-07-26T23:36:22.419 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.419 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.419 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.419 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test watch_notify on pid 41559' 2024-07-26T23:36:22.419 INFO:tasks.workunit.client.0.smithi139.stdout:test watch_notify on pid 41559 2024-07-26T23:36:22.429 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.429 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41559 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.430 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.431 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:++ printf %25s write_operations 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ io: /' 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.432 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.433 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.434 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.435 INFO:tasks.workunit.client.0.smithi139.stderr:+ r=' write_operations' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.436 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.437 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_pool 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41432/exe 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.438 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:++ echo write_operations 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_misc.log 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ misc: /' 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:++ awk '{print $1}' 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_pool.log 2024-07-26T23:36:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ pool: /' 2024-07-26T23:36:22.440 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.440 INFO:tasks.workunit.client.0.smithi139.stderr:+ ff=write_operations 2024-07-26T23:36:22.440 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41601 2024-07-26T23:36:22.440 INFO:tasks.workunit.client.0.smithi139.stderr:+ echo 'test write_operations on pid 41601' 2024-07-26T23:36:22.440 INFO:tasks.workunit.client.0.smithi139.stdout:test write_operations on pid 41601 2024-07-26T23:36:22.470 INFO:tasks.workunit.client.0.smithi139.stderr:+ pids[$f]=41601 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+ ret=0 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' 1 -eq 1 ']' 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40602 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40602 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.471 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.472 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.473 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.474 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.475 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.476 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41388/exe 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -f /etc/bashrc ']' 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+ . /etc/bashrc 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.478 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -z '' ']' 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ BASHRCSOURCED=Y 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ shopt -q login_shell 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.479 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41491/exe 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:+++ umask 2024-07-26T23:36:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' 0022 -eq 0 ']' 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_ec_list 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ SHELL=/bin/bash 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_ec_list.log 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/bash_completion.sh 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.481 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/cobbler.sh 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorgrep.sh 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.482 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.483 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_read_operations 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_read_operations.log 2024-07-26T23:36:22.484 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ read_operations: /' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'grep=grep --color=auto' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'egrep=egrep --color=auto' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorls.sh 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' '!' -t 0 ']' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:+++ return 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.485 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export S_COLORS=auto 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:+++ S_COLORS=auto 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.486 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_misc 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.487 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.488 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.489 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:+++ /usr/libexec/grepconf.sh -c 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.490 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.491 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/debuginfod.sh 2024-07-26T23:36:22.492 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/gawk.sh 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/lang.sh 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ ec_list: /' 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/locale 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.493 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.494 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.495 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41546/exe 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41601/exe 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.496 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.497 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_snapshots 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_write_operations 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_write_operations.log 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG_backup=en_US.UTF-8 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /etc/locale.conf ']' 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -x /usr/bin/sed ']' 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-07-26T23:36:22.498 INFO:tasks.workunit.client.0.smithi139.stderr:++++ export LANG=en_US.UTF-8 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:++++ LANG=en_US.UTF-8 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ LANG=en_US.UTF-8 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ unset LANG_backup config 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n '' ']' 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' dumb = linux ']' 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-07-26T23:36:22.499 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/less.sh 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:++ for i in /etc/profile.d/*.sh 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:++ '[' '' ']' 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:++ . /etc/profile.d/which2.sh 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_snapshots.log 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:+++++ readlink /proc/41559/exe 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ snapshots: /' 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:++++ basename /usr/bin/bash 2024-07-26T23:36:22.500 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_declare='declare -f' 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+++ which_opt=-f 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export which_declare 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+++ export -f which 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset i 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:++ unset -f pathmunge 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ export PATH 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ unset rc 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ ceph_test_neorados_watch_notify 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ watch_notify: /' 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ sed 's/^/ write_operations: /' 2024-07-26T23:36:22.501 INFO:tasks.workunit.client.0.smithi139.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-07-26T23:36:24.035 INFO:tasks.workunit.client.0.smithi139.stdout:x7f6e18076c10 0x7f6e180790d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6e28005770 tx=0x7f6e280056e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.926+0000 7f6e277fe640 10 client.4379.objecter ms_handle_connect 0x7f6e18076c10 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 5238+0+0 (secure 0 0 0) 0x7f6e2c05d810 con 0x55d38d1eb400 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(29..29 src has 1..29) 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map got epochs [29,29] > 0 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding full epoch 29 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e38a3f780 10 librados: wait_for_osdmap done waiting 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e38a3f780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.928+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e3004f070 con 0x55d38d1eb400 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.994+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6e2c05dbd0 con 0x55d38d1eb400 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.994+0000 7f6e277fe640 10 monclient: handle_get_version_reply finishing 1 version 29 2024-07-26T23:36:24.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.994+0000 7f6e34d31640 10 client.4379.objecter _get_latest_version latest 29, have it 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.994+0000 7f6e38a3f780 10 client.4379.objecter create_pool name=ceph_test_rados_api_asio 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.994+0000 7f6e38a3f780 10 client.4379.objecter pool_op_submit 1 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.994+0000 7f6e38a3f780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:21.994+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x55d38d1a72b0 con 0x55d38d1eb400 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v31) ==== 43+0+0 (secure 0 0 0) 0x7f6e2c061c80 con 0x55d38d1eb400 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 pool_op_reply(tid 1 (0) Success v31) 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 10 client.4379.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v31) 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 10 client.4379.objecter have request 1 at 0x55d38d22fc40 Op: create 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 20 client.4379.objecter waiting for client to reach epoch 31 before calling back 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 10 monclient: _renew_subs 2024-07-26T23:36:24.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=30}) -- 0x7f6e1807c8f0 con 0x55d38d1eb400 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.034+0000 7f6e277fe640 10 client.4379.objecter done 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 8 ==== osd_map(30..31 src has 1..31) ==== 7489+0+0 (secure 0 0 0) 0x7f6e2c0617e0 con 0x55d38d1eb400 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(30..31 src has 1..31) 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map got epochs [30,31] > 29 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 30 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 31 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 10 librados: set snap read head -> 1 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 10 librados: writefull oid=exist nspace= 2024-07-26T23:36:24.277 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2c1520 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 31 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x55d38d2c10f0 0x55d38d2ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 2 osd.1 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 2 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 15 client.4379.objecter _send_op 2 to 12.1 on osd.1 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d38d2adaf0 con 0x55d38d2c10f0 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:24.278 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.042+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:24.279 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.108+0000 7f6e36d35640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x55d38d2c10f0 0x55d38d2ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:36:24.279 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.143+0000 7f6e36d35640 10 monclient: get_auth_request con 0x55d38d2c10f0 auth_method 0 2024-07-26T23:36:24.279 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.275+0000 7f6e277fe640 10 client.4379.objecter ms_handle_connect 0x55d38d2c10f0 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.275+0000 api_service: [==========] Running 4 tests from 1 test suite. 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [----------] Global test environment set-up. 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [----------] 4 tests from LibRadosService 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5667 ms) 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ OK ] LibRadosService.RegisterLate (88 ms) 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-07-26T23:36:33.531 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: cluster: 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: id: 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: health: HEALTH_WARN 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 20 pool(s) do not have an application enabled 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: services: 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: osd: 8 osds: 8 up (since 74s), 8 in (since 2m) 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: data: 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: pools: 34 pools, 1033 pgs 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: objects: 4 objects, 449 KiB 2024-07-26T23:36:33.532 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: pgs: 74.734% pgs unknown 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 24.395% pgs not active 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 772 unknown 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 252 creating+peering 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 9 active+clean 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: cluster: 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: id: 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: health: HEALTH_WARN 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 21 pool(s) do not have an application enabled 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: services: 2024-07-26T23:36:33.533 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: osd: 8 osds: 8 up (since 77s), 8 in (since 2m) 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: data: 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: pools: 35 pools, 1065 pgs 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: objects: 4 objects, 449 KiB 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: pgs: 75.493% pgs unknown 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 23.662% pgs not active 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 804 unknown 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 252 creating+peering 2024-07-26T23:36:33.534 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 9 active+clean 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: cluster: 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: id: 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: health: HEALTH_WARN 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 17 pool(s) do not have an application enabled 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: services: 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-07-26T23:36:33.535 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: osd: 8 osds: 8 up (since 79s), 8 in (since 2m) 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: data: 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: pools: 36 pools, 1025 pgs 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: objects: 4 objects, 449 KiB 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-07-26T23:36:33.536 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: pgs: 65.268% pgs unknown 2024-07-26T23:36:34.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 33.561% pgs not activ 7f6e36d35640 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x55d38d2c10f0 0x55d38d2ad5b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:36:34.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.752+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.752+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.752+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:24.752+0000 7f6e267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:36:34.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:25.752+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:25.752+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:25.752+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:26.752+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:26.752+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:26.752+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:26.869+0000 7f6e38798640 10 client.4379.objecter tick 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:27.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:27.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:27.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:28.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:28.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:28.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:29.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:29.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:29.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:30.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:30.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:30.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:31.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:31.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:31.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:31.870+0000 7f6e38798640 10 client.4379.objecter tick 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:32.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:32.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:32.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:33.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:33.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:34.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:33.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:34.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:34.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:34.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:34.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:37.078 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:34.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:37.078 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:34.755+0000 7f6e267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:36:37.078 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:35.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:37.078 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:35.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:35.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.871+0000 7f6e38798640 10 client.4379.objecter tick 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.871+0000 7f6e38798640 2 client.4379.objecter tid 2 on osd.1 is laggy 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.871+0000 7f6e38798640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.871+0000 7f6e38798640 10 monclient: _renew_subs 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.871+0000 7f6e38798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.871+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=32}) -- 0x7f6e08002f90 con 0x55d38d1eb400 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:36.871+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- ping magic: 0 -- 0x7f6e08003360 con 0x55d38d2c10f0 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 9 ==== osd_map(32..39 src has 1..39) ==== 23633+0+0 (secure 0 0 0) 0x7f6e2c061510 con 0x55d38d1eb400 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(32..39 src has 1..39) 2024-07-26T23:36:37.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map got epochs [32,39] > 31 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 32 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 32 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 33 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 33 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.076+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 34 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 34 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:37.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:37.941 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:02:45.613 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:37.970 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:02:45.642 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:38.026 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:02:45.697 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:38.029 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:02:45.700 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:38.228 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:45.890 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:38.261 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:02:45.933 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:38.301 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:02:45.972 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:38.340 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:46.003 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:38.606 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:02:46.268 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:36:40.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 35 2024-07-26T23:36:40.757 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 35 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 36 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 36 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 37 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 37 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 38 2024-07-26T23:36:40.758 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 38 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 39 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 39 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.077+0000 7f6e277fe640 20 client.4379.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.755+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:40.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.755+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:37.755+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:38.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:38.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:38.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:39.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:39.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:39.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:40.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:40.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:43.145 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:40.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:43.145 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:40.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:43.145 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:43.145 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:43.145 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.872+0000 7f6e38798640 10 client.4379.objecter tick 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.872+0000 7f6e38798640 2 client.4379.objecter tid 2 on osd.1 is laggy 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.872+0000 7f6e38798640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.872+0000 7f6e38798640 10 monclient: _renew_subs 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.872+0000 7f6e38798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.872+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=40}) -- 0x7f6e08003560 con 0x55d38d1eb400 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.872+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- ping magic: 0 -- 0x7f6e08003790 con 0x55d38d2c10f0 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 10 ==== osd_map(40..41 src has 1..41) ==== 3208+0+0 (secure 0 0 0) 0x7f6e2c076b90 con 0x55d38d1eb400 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(40..41 src has 1..41) 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map got epochs [40,41] > 39 2024-07-26T23:36:43.146 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 40 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 40 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 41 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 41 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:43.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:43.148 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:41.875+0000 7f6e277fe640 20 client.4379.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-07-26T23:36:43.148 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:42.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:43.148 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:42.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:43.148 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:42.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:43.148 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.143+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 1 ==== osd_map(32..33 src has 1..33) ==== 13651+0+0 (crc 0 0 0) 0x7f6e30058040 con 0x55d38d2c10f0 2024-07-26T23:36:47.120 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.143+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(32..33 src has 1..33) 2024-07-26T23:36:47.120 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.143+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map ignoring epochs [32,33] <= 41 2024-07-26T23:36:47.120 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.143+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:47.120 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.143+0000 7f6e277fe640 20 client.4379.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-07-26T23:36:47.120 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:47.120 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:47.120 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:43.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:44.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:44.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:44.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:44.756+0000 7f6e267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:45.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:45.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:45.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.873+0000 7f6e38798640 10 client.4379.objecter tick 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.873+0000 7f6e38798640 2 client.4379.objecter tid 2 on osd.1 is laggy 2024-07-26T23:36:47.121 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.873+0000 7f6e38798640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.873+0000 7f6e38798640 10 monclient: _renew_subs 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.873+0000 7f6e38798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.873+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=42}) -- 0x7f6e08003960 con 0x55d38d1eb400 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:46.873+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- ping magic: 0 -- 0x7f6e08003bf0 con 0x55d38d2c10f0 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 11 ==== osd_map(42..43 src has 1..43) ==== 3045+0+0 (secure 0 0 0) 0x7f6e2c076d90 con 0x55d38d1eb400 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(42..43 src has 1..43) 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map got epochs [42,43] > 41 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 42 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 42 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:47.122 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 43 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 10 client.4379.objecter checking op 2 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 43 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.118+0000 7f6e277fe640 20 client.4379.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:47.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:48.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:51.910 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:48.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:48.756+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:49.756+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:49.756+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:49.757+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:50.757+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:50.757+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:50.757+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.757+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.757+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.757+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.874+0000 7f6e38798640 10 client.4379.objecter tick 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.874+0000 7f6e38798640 2 client.4379.objecter tid 2 on osd.1 is laggy 2024-07-26T23:36:51.911 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.874+0000 7f6e38798640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.874+0000 7f6e38798640 10 monclient: _renew_subs 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.874+0000 7f6e38798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.874+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=44}) -- 0x7f6e08003960 con 0x55d38d1eb400 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.874+0000 7f6e38798640 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- ping magic: 0 -- 0x7f6e08003e20 con 0x55d38d2c10f0 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 2 ==== osd_op_reply(2 exist [writefull 0~5] v33'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(2 exist [writefull 0~5] v33'1 uv1 ondisk = 0) 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 2 ondisk uv 1 in 12.1 attempt 0 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 2 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 15 client.4379.objecter _finish_op 2 2024-07-26T23:36:51.912 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 2 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 10 librados: Objecter returned from writefull r=0 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2adee0 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 43 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 3 osd.1 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 3 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 15 client.4379.objecter _send_op 3 to 12.1 on osd.1 2024-07-26T23:36:51.913 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:3 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x55d38d2c1520 con 0x55d38d2c10f0 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 43 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 4 osd.1 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 4 2024-07-26T23:36:51.914 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 15 client.4379.objecter _send_op 4 to 12.1 on osd.1 2024-07-26T23:36:52.161 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:4 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x55d38d2b3f90 con 0x55d38d2c10f0 2024-07-26T23:36:52.161 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:52.161 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:51.908+0000 7f6e38a3f780 5 client.4379.objecter 2 in flight 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 3 ondisk uv 1 in 12.1 attempt 0 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 5 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 3 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 15 client.4379.objecter _finish_op 3 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 3 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 5 client.4379.objecter 1 in flight 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:52.162 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 4 ondisk uv 0 in 12.1 attempt 0 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 4 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 15 client.4379.objecter _finish_op 4 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 4 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.159+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (252 ms) 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 43 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:52.163 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 5 osd.1 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 5 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 15 client.4379.objecter _send_op 5 to 12.1 on osd.1 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:5 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x55d38d2adee0 con 0x55d38d2c10f0 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b4360 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 43 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 6 osd.1 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 6 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 15 client.4379.objecter _send_op 6 to 12.1 on osd.1 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:6 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x55d38d370ef0 con 0x55d38d2c10f0 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.160+0000 7f6e38a3f780 5 client.4379.objecter 2 in flight 2024-07-26T23:36:52.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 5 ondisk uv 1 in 12.1 attempt 0 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 5 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 5 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 15 client.4379.objecter _finish_op 5 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 5 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 5 client.4379.objecter 1 in flight 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:52.245 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 6 ondisk uv 0 in 12.1 attempt 0 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 6 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 15 client.4379.objecter _finish_op 6 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 6 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (81 ms) 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 43 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:52.246 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 7 osd.1 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 7 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 15 client.4379.objecter _send_op 7 to 12.1 on osd.1 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:7 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x55d38d2b4360 con 0x55d38d2c10f0 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.247 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:52.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d3b12b0 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 43 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 8 osd.1 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 8 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 15 client.4379.objecter _send_op 8 to 12.1 on osd.1 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:8 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) -- 0x55d38d3b1760 con 0x55d38d2c10f0 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:52.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.241+0000 7f6e38a3f780 5 client.4379.objecter 2 in flight 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.757+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.757+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.757+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 7 ondisk uv 1 in 12.1 attempt 0 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 5 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 7 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 15 client.4379.objecter _finish_op 7 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:52.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 7 2024-07-26T23:36:52.772 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 5 client.4379.objecter 1 in flight 2024-07-26T23:36:52.772 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:52.772 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:36:53.639 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768e 2024-07-26T23:36:53.639 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 669 unknown 2024-07-26T23:36:53.639 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 293 creating+peering 2024-07-26T23:36:53.639 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 51 creating+activating 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 12 active+clean 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5919 ms) 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ RUN ] LibRadosService.Status 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ OK ] LibRadosService.Status (20071 ms) 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [----------] 4 tests from LibRadosService (31745 ms total) 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [----------] Global test environment tear-down 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31745 ms total) 2024-07-26T23:36:53.640 INFO:tasks.workunit.client.0.smithi139.stdout: api_service: [ PASSED ] 4 tests. 2024-07-26T23:36:54.759 INFO:tasks.workunit.client.0.smithi139.stdout:+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:54.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 8 ondisk uv 0 in 12.1 attempt 0 2024-07-26T23:36:54.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:36:54.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:54.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 8 2024-07-26T23:36:54.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 15 client.4379.objecter _finish_op 8 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 8 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (527 ms) 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 librados: queue_aio_write 0x55d38d15e4d0 completion 0x55d38d2c18c0 write_seq 1 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:54.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 43 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 9 osd.1 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 9 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 15 client.4379.objecter _send_op 9 to 12.1 on osd.1 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:9 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) -- 0x55d38d3722a0 con 0x55d38d2c10f0 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:52.768+0000 7f6e38a3f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-07-26T23:36:54.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:53.757+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:54.762 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:53.757+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:54.762 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:53.757+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:54.762 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:54.757+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:54.762 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:54.757+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:55.426 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:54.757+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-07-26T23:36:55.426 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [----------] Global test environment set-up. 2024-07-26T23:36:55.426 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-07-26T23:36:55.426 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-07-26T23:36:55.426 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5526 ms) 2024-07-26T23:36:55.426 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (205 ms) 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20098 ms) 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: attempt 0 of 20 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (7687 ms) 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (33516 ms total) 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [----------] Global test environment tear-down 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (33516 ms total) 2024-07-26T23:36:55.427 INFO:tasks.workunit.client.0.smithi139.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-07-26T23:36:55.537 INFO:tasks.workunit.client.0.smithi139.stdout:.admin 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:54.757+0000 7f6e267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 9 ==== osd_map(44..44 src has 1..44) ==== 1962+0+0 (crc 0 0 0) 0x7f6e30066440 con 0x55d38d2c10f0 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(44..44 src has 1..44) 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map got epochs [44,44] > 43 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map decoding incremental epoch 44 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 10 client.4379.objecter checking op 9 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:55.538 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.103+0000 7f6e277fe640 20 client.4379.objecter 9 12.d3a59201 osd.1 exist [write 0~5 in=5b] 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 10 ==== osd_op_reply(9 exist [write 0~5] v44'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(9 exist [write 0~5] v44'2 uv2 ondisk = 0) 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 9 ondisk uv 2 in 12.1 attempt 0 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 9 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 15 client.4379.objecter _finish_op 9 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 9 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e36d35640 20 librados: complete_aio_write 0x55d38d2c18c0 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2768 ms) 2024-07-26T23:36:55.539 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-07-26T23:36:55.540 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-07-26T23:36:55.540 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 librados: queue_aio_write 0x55d38d15e4d0 completion 0x55d38d372690 write_seq 2 2024-07-26T23:36:55.540 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:55.540 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:55.540 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:55.540 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:55.643 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:55.643 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 10 osd.1 2024-07-26T23:36:55.643 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:55.643 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 10 2024-07-26T23:36:55.643 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 15 client.4379.objecter _send_op 10 to 12.1 on osd.1 2024-07-26T23:36:55.644 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:10 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) -- 0x55d38d3722a0 con 0x55d38d2c10f0 2024-07-26T23:36:55.644 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:55.644 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:55.644 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.536+0000 7f6e38a3f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 11 ==== osd_op_reply(10 exist [write 0~5] v44'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(10 exist [write 0~5] v44'3 uv3 ondisk = 0) 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 10 ondisk uv 3 in 12.1 attempt 0 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 10 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 15 client.4379.objecter _finish_op 10 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 10 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e36d35640 20 librados: complete_aio_write 0x55d38d372690 2024-07-26T23:36:55.645 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (105 ms) 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 20 librados: queue_aio_write 0x55d38d15e4d0 completion 0x55d38d2b4990 write_seq 3 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:55.646 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:55.877 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 11 osd.1 2024-07-26T23:36:55.879 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 11 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 15 client.4379.objecter _send_op 11 to 12.1 on osd.1 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:11 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) -- 0x55d38d3947c0 con 0x55d38d2c10f0 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.641+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.642+0000 7f6e38a3f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.757+0000 7f6e267fc640 10 monclient: tick 2024-07-26T23:36:55.880 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.757+0000 7f6e267fc640 10 monclient: _check_auth_tickets 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.757+0000 7f6e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 12 ==== osd_op_reply(11 exist [write 0~5] v44'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(11 exist [write 0~5] v44'4 uv4 ondisk = 0) 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 11 ondisk uv 4 in 12.1 attempt 0 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 11 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 15 client.4379.objecter _finish_op 11 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 11 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e36d35640 20 librados: complete_aio_write 0x55d38d2b4990 2024-07-26T23:36:55.881 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (234 ms) 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 12 osd.1 2024-07-26T23:36:55.882 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.209 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 12 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 15 client.4379.objecter _send_op 12 to 12.1 on osd.1 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:12 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371290 con 0x55d38d2c10f0 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d371740 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.210 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 13 osd.1 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 13 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 15 client.4379.objecter _send_op 13 to 12.1 on osd.1 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:13 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371bf0 con 0x55d38d2c10f0 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:55.875+0000 7f6e38a3f780 5 client.4379.objecter 2 in flight 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.077+0000 7f6e277fe640 1 -- 172.21.15.139:0/1617108442 <== mon.2 v2:172.21.15.139:3301/0 12 ==== osd_map(44..44 src has 1..44) ==== 1962+0+0 (secure 0 0 0) 0x7f6e2c076d90 con 0x55d38d1eb400 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.077+0000 7f6e277fe640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_map(44..44 src has 1..44) 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.077+0000 7f6e277fe640 3 client.4379.objecter handle_osd_map ignoring epochs [44,44] <= 44 2024-07-26T23:36:56.211 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.077+0000 7f6e277fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.077+0000 7f6e277fe640 20 client.4379.objecter 12 12.d3a59201 osd.1 exist [read 0~0] 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.077+0000 7f6e277fe640 20 client.4379.objecter 13 12.a7bbaa1 osd.1 noexist [read 0~0] 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 13 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 12 ondisk uv 4 in 12.1 attempt 0 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 5 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 12 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 15 client.4379.objecter _finish_op 12 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 12 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.212 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 14 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 13 ondisk uv 0 in 12.1 attempt 0 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 13 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 15 client.4379.objecter _finish_op 13 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 13 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (332 ms) 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-07-26T23:36:56.213 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 14 osd.1 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 14 2024-07-26T23:36:56.214 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 15 client.4379.objecter _send_op 14 to 12.1 on osd.1 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:14 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371290 con 0x55d38d2c10f0 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d371740 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 15 osd.1 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.226 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 15 2024-07-26T23:36:56.227 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 15 client.4379.objecter _send_op 15 to 12.1 on osd.1 2024-07-26T23:36:56.227 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:15 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371bf0 con 0x55d38d2c10f0 2024-07-26T23:36:56.227 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:56.227 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.207+0000 7f6e38a3f780 5 client.4379.objecter 2 in flight 2024-07-26T23:36:56.227 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 15 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.227 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-07-26T23:36:56.227 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.228 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 14 ondisk uv 4 in 12.1 attempt 0 2024-07-26T23:36:56.228 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 5 2024-07-26T23:36:56.228 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 14 2024-07-26T23:36:56.228 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 15 client.4379.objecter _finish_op 14 2024-07-26T23:36:56.228 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:56.228 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 14 2024-07-26T23:36:56.228 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 16 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 15 ondisk uv 0 in 12.1 attempt 0 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 15 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 15 client.4379.objecter _finish_op 15 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 15 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (17 ms) 2024-07-26T23:36:56.229 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 16 osd.1 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 16 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 15 client.4379.objecter _send_op 16 to 12.1 on osd.1 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:16 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371290 con 0x55d38d2c10f0 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.230 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.231 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d371740 2024-07-26T23:36:56.231 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-07-26T23:36:56.231 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-07-26T23:36:56.231 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.231 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.231 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 17 osd.1 2024-07-26T23:36:56.251 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.252 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 17 2024-07-26T23:36:56.252 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 15 client.4379.objecter _send_op 17 to 12.1 on osd.1 2024-07-26T23:36:56.252 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:17 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371bf0 con 0x55d38d2c10f0 2024-07-26T23:36:56.252 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:56.252 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.224+0000 7f6e38a3f780 5 client.4379.objecter 2 in flight 2024-07-26T23:36:56.253 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 17 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.253 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-07-26T23:36:56.253 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.253 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 16 ondisk uv 4 in 12.1 attempt 0 2024-07-26T23:36:56.253 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 5 2024-07-26T23:36:56.253 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 16 2024-07-26T23:36:56.254 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 15 client.4379.objecter _finish_op 16 2024-07-26T23:36:56.254 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 5 2024-07-26T23:36:56.254 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 16 2024-07-26T23:36:56.254 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.255 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 18 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.255 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-07-26T23:36:56.255 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.255 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 17 ondisk uv 0 in 12.1 attempt 0 2024-07-26T23:36:56.255 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 10 client.4379.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-07-26T23:36:56.255 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:56.255 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 17 2024-07-26T23:36:56.256 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 15 client.4379.objecter _finish_op 17 2024-07-26T23:36:56.256 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.256 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 17 2024-07-26T23:36:56.256 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:56.256 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (26 ms) 2024-07-26T23:36:56.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-07-26T23:36:56.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 20 librados: queue_aio_write 0x55d38d15e4d0 completion 0x55d38d372140 write_seq 4 2024-07-26T23:36:56.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 18 osd.1 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 18 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 15 client.4379.objecter _send_op 18 to 12.1 on osd.1 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:18 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371290 con 0x55d38d2c10f0 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.250+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 19 ==== osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 18 ondisk uv 5 in 12.1 attempt 0 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 18 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 15 client.4379.objecter _finish_op 18 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 18 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e36d35640 20 librados: complete_aio_write 0x55d38d372140 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (149 ms) 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 20 librados: queue_aio_write 0x55d38d15e4d0 completion 0x55d38d3948d0 write_seq 5 2024-07-26T23:36:56.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 ba api_misc: [==========] Running 12 tests from 4 test suites. 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] Global test environment set-up. 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-07-26T23:36:56.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2024-07-26T23:36:22.081+0000 7f75427eb780 -1 failed for service _ceph-mon._tcp 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: ceph-mon 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: 2024-07-26T23:36:22.083+0000 7f75427eb780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (369 ms) 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5022 ms) 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5391 ms total) 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-07-26T23:36:56.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x5630543de3b0 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053a57340 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053a62f40 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x5630544097d0 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x5630543c1a40 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x5630537a8f70 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bcdb60 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bce580 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bcef80 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bcf5e0 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bd0050 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bced40 2024-07-26T23:36:56.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x5630544807a0 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bcfd90 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x5630544815d0 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054480580 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054482560 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054481330 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054483410 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054482240 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054484310 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054484590 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054483120 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bd1970 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563053bd23e0 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 0x563054484050 2024-07-26T23:36:56.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: started 26 aios 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x5630543de3b0 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053a57340 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053a62f40 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x5630544097d0 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x5630543c1a40 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x5630537a8f70 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bcdb60 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bce580 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bcef80 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bcf5e0 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bd0050 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bced40 2024-07-26T23:36:56.407 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x5630544807a0 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bcfd90 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x5630544815d0 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054480580 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054482560 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054481330 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054483410 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054482240 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054484310 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054484590 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054483120 2024-07-26T23:36:56.408 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563053bd1970 2024-07-26T23:36:56.468 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x56se exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:56.495 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 19 osd.1 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 19 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 15 client.4379.objecter _send_op 19 to 12.1 on osd.1 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:19 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371290 con 0x55d38d2c10f0 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.399+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 20 ==== osd_op_reply(19 exist [writefull 0~5] v44'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(19 exist [writefull 0~5] v44'6 uv6 ondisk = 0) 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.496 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 19 ondisk uv 6 in 12.1 attempt 0 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 19 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 15 client.4379.objecter _finish_op 19 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 19 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e36d35640 20 librados: complete_aio_write 0x55d38d3948d0 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (68 ms) 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 20 librados: queue_aio_write 0x55d38d15e4d0 completion 0x55d38d394a00 write_seq 6 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 10 client.4379.objecter _op_submit op 0x55d38d2b3ae0 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 20 client.4379.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 20 client.4379.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-07-26T23:36:56.497 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 10 client.4379.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-07-26T23:36:56.562 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 20 client.4379.objecter _get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 10 client.4379.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 20 osd.1 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 20 client.4379.objecter get_session s=0x55d38d2c2830 osd=1 3 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 15 client.4379.objecter _session_op_assign 1 20 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 15 client.4379.objecter _send_op 20 to 12.1 on osd.1 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- osd_op(unknown.0.0:20 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) -- 0x55d38d371290 con 0x55d38d2c10f0 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.467+0000 7f6e38a3f780 5 client.4379.objecter 1 in flight 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 <== osd.1 v2:172.21.15.139:6800/3107328002 21 ==== osd_op_reply(20 exist [writefull 0~5] v44'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6e3005b040 con 0x55d38d2c10f0 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 10 client.4379.objecter ms_dispatch 0x55d38d197990 osd_op_reply(20 exist [writefull 0~5] v44'7 uv7 ondisk = 0) 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 10 client.4379.objecter in handle_osd_op_reply 2024-07-26T23:36:56.563 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 7 client.4379.objecter handle_osd_op_reply 20 ondisk uv 7 in 12.1 attempt 0 2024-07-26T23:36:56.564 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 10 client.4379.objecter op 0 rval 0 len 0 2024-07-26T23:36:56.564 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 15 client.4379.objecter handle_osd_op_reply completed tid 20 2024-07-26T23:36:56.564 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 15 client.4379.objecter _finish_op 20 2024-07-26T23:36:56.564 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 4 2024-07-26T23:36:56.564 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 15 client.4379.objecter _session_op_remove 1 20 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 5 client.4379.objecter 0 in flight 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 20 librados: complete_aio_write 0x55d38d394a00 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (94 ms) 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 10 librados: watch_flush enter 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 10 librados: watch_flush exit 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 10 client.4379.objecter close_session for osd.1 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x55d38d2c10f0 msgr2=0x55d38d2ad5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:36:56.565 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x55d38d2c10f0 0x55d38d2ad5b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:36:56.566 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 20 client.4379.objecter put_session s=0x55d38d2c2830 osd=1 1 2024-07-26T23:36:56.566 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 20 client.4379.objecter shutdown clearing up homeless session... 2024-07-26T23:36:56.566 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 10 client.4379.objecter successfully canceled tick 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6e18076c10 msgr2=0x7f6e180790d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6e18076c10 0x7f6e180790d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6e28005770 tx=0x7f6e280056e0 comp rx=0 tx=0).stop 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 10 monclient: shutdown 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 msgr2=0x55d38d2b05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 --2- 172.21.15.139:0/1617108442 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x55d38d1eb400 0x55d38d2b05a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00aa90 tx=0x7f6e2c002fd0 comp rx=0 tx=0).stop 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e36d35640 1 -- 172.21.15.139:0/1617108442 reap_dead start 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 shutdown_connections 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 >> 172.21.15.139:0/1617108442 conn(0x55d38d230760 msgr2=0x55d38d2290d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:36:56.567 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 shutdown_connections 2024-07-26T23:36:56.568 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 -- 172.21.15.139:0/1617108442 wait complete. 2024-07-26T23:36:56.568 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.561+0000 7f6e38a3f780 1 librados: shutdown 2024-07-26T23:36:56.568 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [----------] 12 tests from AsioRados (4653 ms total) 2024-07-26T23:36:56.568 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: 2024-07-26T23:36:56.568 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [----------] Global test environment tear-down 2024-07-26T23:36:56.568 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (34853 ms total) 2024-07-26T23:36:56.568 INFO:tasks.workunit.client.0.smithi139.stdout: api_asio: [ PASSED ] 12 tests. 2024-07-26T23:37:07.399 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] Global test environment set-up. 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] 7 tests from LibRadosList 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosList.ListObjects (30753 ms) 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1058 ms) 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset foo1,foo2,foo3 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo1 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo2 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo3 2024-07-26T23:37:07.400 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset foo1,foo4,foo5 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo4 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo5 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo1 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset foo6,foo7 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo7 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo6 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns1:foo4 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns1:foo5 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns2:foo7 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns2:foo6 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns1:foo1 2024-07-26T23:37:07.401 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: :foo1 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: :foo2 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: :foo3 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (2334 ms) 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 1 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 10 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 13 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 7 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 14 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 0 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 15 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 11 0 2024-07-26T23:37:07.402 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 5 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 8 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 6 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 3 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 4 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 12 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 9 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 2 0 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (1849 ms) 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: x cursor=MIN 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2024-07-26T23:37:07.403 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2024-07-26T23:37:07.404 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: FIRST> seek to MIN oid=1 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:566253c9:::13:head 2024-07-26T23:37:07.405 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-07-26T23:37:07.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2024-07-26T23:37:07.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-07-26T23:37:07.406 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-07-26T23:37:09.079 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=14 cursor=7:62a1935d:::14:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:863748b0:::15:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-07-26T23:37:09.080 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-07-26T23:37:09.081 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:e960b815:::9:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-07-26T23:37:09.082 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=10 expected=10 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=1 expected=1 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:566253c9:::13:head 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=13 expected=13 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-07-26T23:37:09.083 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=3 expected=3 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=6 expected=6 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=8 expected=8 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=5 expected=5 2024-07-26T23:37:09.084 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-07-26T23:37:09.085 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2024-07-26T23:37:09.085 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-07-26T23:37:09.085 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=4 expected=4 2024-07-26T23:37:09.085 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-07-26T23:37:10.229 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:10.222+0000 12a66640 -1 osd.0 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:11.278 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:11.274+0000 12a66640 -1 osd.0 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:12.290 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:12.289+0000 12a66640 -1 osd.0 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: ap handler_error: Running main() from gmock_main.cc 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [----------] Global test environment set-up. 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (50915 ms) 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [----------] 1 test from neocls_handler_error (50915 ms total) 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [----------] Global test environment tear-down 2024-07-26T23:37:12.958 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [==========] 1 test from 1 test suite ran. (50915 ms total) 2024-07-26T23:37:12.959 INFO:tasks.workunit.client.0.smithi139.stdout: handler_error: [ PASSED ] 1 test. 2024-07-26T23:37:13.167 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-07-26T23:37:13.167 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-07-26T23:37:13.167 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-07-26T23:37:13.167 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-07-26T23:37:13.167 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-07-26T23:37:13.167 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (1200 ms) 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (87 ms) 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (50006 ms) 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (51293 ms total) 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (51293 ms total) 2024-07-26T23:37:13.168 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-07-26T23:37:13.218 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-07-26T23:37:13.218 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-07-26T23:37:13.218 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-07-26T23:37:13.218 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (28616 ms) 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (224 ms) 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (185 ms) 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (567 ms) 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (166 ms) 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-07-26T23:37:13.219 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (187 ms) 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (76 ms) 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (157 ms) 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (139 ms) 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (134 ms) 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (106 ms) 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (350 ms) 2024-07-26T23:37:13.220 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (408 ms) 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (348 ms) 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (203 ms) 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (91 ms) 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (31957 ms total) 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (51280 ms total) 2024-07-26T23:37:13.221 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-07-26T23:37:14.687 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:14.502+0000 12a66640 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:15.826 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:15.735+0000 12a66640 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50.a316516c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:16.780 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:16.753+0000 12a66640 -1 osd.0 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:16.980 INFO:tasks.workunit.client.0.smithi139.stdout: cls: Running main() from gmock_main.cc 2024-07-26T23:37:16.980 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-07-26T23:37:16.980 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [----------] Global test environment set-up. 2024-07-26T23:37:16.980 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [----------] 1 test from NeoRadosCls 2024-07-26T23:37:16.980 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-07-26T23:37:16.981 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [ OK ] NeoRadosCls.DNE (54968 ms) 2024-07-26T23:37:16.981 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [----------] 1 test from NeoRadosCls (54968 ms total) 2024-07-26T23:37:16.981 INFO:tasks.workunit.client.0.smithi139.stdout: cls: 2024-07-26T23:37:16.981 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [----------] Global test environment tear-down 2024-07-26T23:37:16.981 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [==========] 1 test from 1 test suite ran. (54969 ms total) 2024-07-26T23:37:16.981 INFO:tasks.workunit.client.0.smithi139.stdout: cls: [ PASSED ] 1 test. 2024-07-26T23:37:17.918 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:17.899+0000 12a66640 -1 osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:18.135 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: Running main() from gmock_main.cc 2024-07-26T23:37:18.135 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-07-26T23:37:18.135 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [----------] Global test environment set-up. 2024-07-26T23:37:18.135 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-07-26T23:37:18.135 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-07-26T23:37:18.135 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1010 ms) 2024-07-26T23:37:18.135 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (97 ms) 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (50190 ms) 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:36:53.442881+0000 mon.a [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:03.048621+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi139-41526-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi139-41526-2"}]': finished 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:08.167947+0000 mon.a [WRN] Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:10.227608+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'LongNamesmithi139-41590-3' : 1 ]) 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:11.275600+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'LongNamesmithi139-41590-3' : 1 ]) 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:12.290267+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'LongNamesmithi139-41590-3' : 1 ]) 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:13.170445+0000 mon.b [INF] from='client.? 172.21.15.139:0/3077603204' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-40815-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:13.288740+0000 mon.c [INF] from='client.? 172.21.15.139:0/2928656222' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-07-26T23:37:18.136 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:13.308377+0000 mon.b [INF] from='client.? 172.21.15.139:0/17341968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi139-40937-7"} : dispatch 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:13.422442+0000 mon.b [INF] from='client.? 172.21.15.139:0/4292942050' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi139-40654-10"} : dispatch 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.355733+0000 mon.a [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec, osd.0 has slow ops (SLOW_OPS) 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.378959+0000 mon.a [WRN] pool 'PoolQuotaPP_smithi139-40535-3' is full (reached quota's max_bytes: 4 KiB) 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.471211+0000 mon.a [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.503760+0000 osd.0 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'LongNamesmithi139-41590-3' : 1 ]) 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.528387+0000 mon.c [INF] from='client.? 172.21.15.139:0/371563819' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi139-40552-16"} : dispatch 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.657187+0000 client.admin [INF] onexx 2024-07-26T23:37:18.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.713995+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-07-26T23:37:19.366 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:19.162+0000 12a66640 -1 osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4738.0:2 50.c 50:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-07-26T23:37:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.798441+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi139-40552-16"} : dispatch 2024-07-26T23:37:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.943161+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-40815-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.991037+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi139-40937-7"} : dispatch 2024-07-26T23:37:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:14.998617+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi139-40654-10"} : dispatch 2024-07-26T23:37:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.598359+0000 mon.b [INF] from='client.? 172.21.15.139:0/17341968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi139-40937-7"} : dispatch 2024-07-26T23:37:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.607811+0000 mon.b [INF] from='client.? 172.21.15.139:0/4292942050' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi139-40654-10"} : dispatch 2024-07-26T23:37:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.671885+0000 mon.c [INF] from='client.? 172.21.15.139:0/475303739' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi139-40934-7"} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.693553+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi139-40937-7"} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.708443+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi139-40654-10"} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.730777+0000 mon.b [INF] from='client.? 172.21.15.139:0/17341968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi139-40937-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.737260+0000 mon.b [INF] from='client.? 172.21.15.139:0/4292942050' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi139-40654-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.748134+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi139-40937-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.763531+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi139-40654-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.831559+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:16.901376+0000 mon.c [INF] from='client.? 172.21.15.139:0/3757504916' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi139-41023-1"} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:17.088616+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi139-40934-7"} : dispatch 2024-07-26T23:37:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:17.139737+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi139-41023-1"} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.054806+0000 mon.b [INF] from='client.? 172.21.15.139:0/3077603204' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-40815-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.284310+0000 mon.c [INF] from='client.? 172.21.15.139:0/2551207869' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi139-41614-1"} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.350253+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi139-41614-1"} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.354946+0000 mon.c [INF] from='client.? 172.21.15.139:0/3697591072' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-41136-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.359500+0000 mon.c [INF] from='client.? 172.21.15.139:0/1185533100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi139-40984-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.499905+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-40815-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.544530+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-41136-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.549683+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi139-40984-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.579317+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.969931+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi139-40937-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi139-40937-7"}]': finished 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.985575+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi139-40654-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi139-40654-10"}]': finished 2024-07-26T23:37:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.988981+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi139-40934-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-07-26T23:37:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.992257+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi139-40559-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-07-26T23:37:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.995453+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi139-40552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-07-26T23:37:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.997643+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-40762-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-07-26T23:37:23.059 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:20.999892+0000 mon.a [INF] from='client.? 172.21.15.139:0/1843600157' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi139-40668-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-07-26T23:37:23.059 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.001749+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi139-41614-1"}]': finished 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.003449+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-40815-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.005433+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi139-41136-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.007444+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi139-40984-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.437767+0000 mon.c [INF] from='client.? 172.21.15.139:0/371563819' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi139-40552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi139-40552-16"} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.464927+0000 mon.c [INF] from='client.? 172.21.15.139:0/475303739' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi139-40934-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi139-40934-7"} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.517413+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi139-40552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi139-40552-16"} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.460525+0000 mon.b [INF] from='client.? 172.21.15.139:0/3077603204' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi139-40815-4", "tierpool": "test-rados-api-smithi139-40815-4-cache"} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.469527+0000 mon.b [INF] from='client.? 172.21.15.139:0/2339164814' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi139-40559-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi139-40559-24"} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.478980+0000 mon.b [INF] from='client.? 172.21.15.139:0/333553032' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi139-41023-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.491989+0000 mon.c [INF] from='client.? 172.21.15.139:0/1185533100' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi139-40984-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi139-40984-12"} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.541319+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi139-40934-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi139-40934-7"} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.559235+0000 mon.b [INF] from='client.? 172.21.15.139:0/1873175213' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi139-41614-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:23.060 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.765065+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi139-40815-4", "tierpool": "test-rados-api-smithi139-40815-4-cache"} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.847894+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi139-40559-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi139-40559-24"} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.859332+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi139-40984-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi139-40984-12"} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.877084+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi139-41023-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:21.885479+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi139-41614-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.362280+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi139-40635-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi139-40635-10"}]': finished 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.380109+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi139-40815-4", "tierpool": "test-rados-api-smithi139-40815-4-cache"}]': finished 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.382884+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi139-41023-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.385878+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi139-41614-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.739209+0000 mon.b [INF] from='client.? 172.21.15.139:0/1873175213' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi139-41614-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi139-41614-2"} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.748087+0000 mon.b [INF] from='client.? 172.21.15.139:0/3077603204' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi139-40815-4", "overlaypool": "test-rados-api-smithi139-40815-4-cache"} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.763379+0000 mon.c [INF] from='client.? 172.21.15.139:0/2928656222' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-07-26T23:37:23.061 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.775703+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi139-41614-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi139-41614-2"} : dispatch 2024-07-26T23:37:25.467 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.787274+0000 mon.a [INF] from='client.? ' enti pool: Running main() from gmock_main.cc 2024-07-26T23:37:25.467 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [----------] Global test environment set-up. 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ OK ] NeoRadosPools.PoolList (6900 ms) 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (6851 ms) 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (7991 ms) 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (36204 ms) 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2738 ms) 2024-07-26T23:37:25.468 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-07-26T23:37:25.469 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2685 ms) 2024-07-26T23:37:25.469 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [----------] 6 tests from NeoRadosPools (63369 ms total) 2024-07-26T23:37:25.469 INFO:tasks.workunit.client.0.smithi139.stdout: pool: 2024-07-26T23:37:25.469 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [----------] Global test environment tear-down 2024-07-26T23:37:25.469 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [==========] 6 tests from 1 test suite ran. (63369 ms total) 2024-07-26T23:37:25.469 INFO:tasks.workunit.client.0.smithi139.stdout: pool: [ PASSED ] 6 tests. 2024-07-26T23:37:26.302 INFO:tasks.workunit.client.0.smithi139.stdout:ty='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi139-40815-4", "overlaypool": "test-rados-api-smithi139-40815-4-cache"} : dispatch 2024-07-26T23:37:26.302 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.819261+0000 client.admin [INF] threexx 2024-07-26T23:37:26.302 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: got: 2024-07-26T23:37:22.826011+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-07-26T23:37:26.302 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (13139 ms) 2024-07-26T23:37:26.302 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (64436 ms total) 2024-07-26T23:37:26.303 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: 2024-07-26T23:37:26.303 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [----------] Global test environment tear-down 2024-07-26T23:37:26.303 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (64441 ms total) 2024-07-26T23:37:26.303 INFO:tasks.workunit.client.0.smithi139.stdout: api_cmd: [ PASSED ] 4 tests. 2024-07-26T23:37:30.341 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:03:38.012 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.421 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:03:38.093 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.468 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:03:38.116 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.482 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:03:38.153 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.500 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:38.162 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.502 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:38.165 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.654 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:03:38.325 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.712 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:03:38.374 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:30.739 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:03:38.411 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:50.773 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: Running main() from gmock_main.cc 2024-07-26T23:37:50.773 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-07-26T23:37:50.773 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [----------] Global test environment set-up. 2024-07-26T23:37:50.773 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-07-26T23:37:50.773 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (17207 ms) 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (71502 ms) 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (88709 ms total) 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [----------] Global test environment tear-down 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (88709 ms total) 2024-07-26T23:37:50.774 INFO:tasks.workunit.client.0.smithi139.stdout: ec_io: [ PASSED ] 2 tests. 2024-07-26T23:37:50.937 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:50.918+0000 12a66640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4831.0:2 64.6 64.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e49) 2024-07-26T23:37:52.282 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:52.246+0000 12a66640 -1 osd.0 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4831.0:2 64.6 64.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e49) 2024-07-26T23:37:53.374 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:53.259+0000 12a66640 -1 osd.0 70 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4831.0:2 64.6 64.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e49) 2024-07-26T23:37:54.679 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:54.676+0000 12a66640 -1 osd.0 70 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4831.0:2 64.6 64:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49) 2024-07-26T23:37:55.702 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:55.699+0000 12a66640 -1 osd.0 71 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4831.0:2 64.6 64:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49) 2024-07-26T23:37:56.279 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:04:03.950 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.289 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:04:03.961 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.377 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:04.039 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.388 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:04.051 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.391 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:04:04.063 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.414 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:04:04.086 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.475 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:04:04.147 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.623 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:04.285 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.689 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:04:04.360 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:37:56.954 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:37:56.814+0000 12a66640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4831.0:2 64.6 64:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49) 2024-07-26T23:38:05.172 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: Running main() from gmock_main.cc 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [----------] Global test environment set-up. 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7172 ms) 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4228 ms) 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (6032 ms) 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (35403 ms) 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-07-26T23:38:05.173 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (9537 ms) 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6202 ms) 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6558 ms) 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (12754 ms) 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (15556 ms) 2024-07-26T23:38:05.174 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [----------] 9 tests from LibRadosPools (103442 ms total) 2024-07-26T23:38:05.175 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: 2024-07-26T23:38:05.175 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [----------] Global test environment tear-down 2024-07-26T23:38:05.175 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (103442 ms total) 2024-07-26T23:38:05.175 INFO:tasks.workunit.client.0.smithi139.stdout: api_pool: [ PASSED ] 9 tests. 2024-07-26T23:38:05.727 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: Running main() from gmock_main.cc 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [----------] Global test environment set-up. 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: seed 40559 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (26018 ms) 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (3488 ms) 2024-07-26T23:38:05.728 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (429 ms) 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (14434 ms) 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (44 ms) 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (56 ms) 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (48 ms) 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (64 ms) 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (97 ms) 2024-07-26T23:38:05.729 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (306 ms) 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (51 ms) 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (72 ms) 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (85 ms) 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (85 ms) 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (70 ms) 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-07-26T23:38:05.730 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (304 ms) 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (97 ms) 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (88 ms) 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (71 ms) 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (42 ms) 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (2220 ms) 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (48169 ms total) 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-07-26T23:38:05.731 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-07-26T23:38:05.732 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (32170 ms) 2024-07-26T23:38:05.732 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-07-26T23:38:05.732 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (3370 ms) 2024-07-26T23:38:05.732 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.Sparse watch_notify: Running main() from gmock_main.cc 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [----------] Global test environment set-up. 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: handle_notify cookie 94593929555568 notify_id 184683593728 notifier_gid 4454 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (51051 ms) 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: Trying... 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: handle_notify cookie 94593942794960 notify_id 313532612609 notifier_gid 4819 2024-07-26T23:38:07.570 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: Waiting for 3.000000000s 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: Timed out. 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: Flushing... 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: Flushed... 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (54376 ms) 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (105427 ms total) 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [----------] Global test environment tear-down 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (105427 ms total) 2024-07-26T23:38:07.571 INFO:tasks.workunit.client.0.smithi139.stdout: watch_notify: [ PASSED ] 2 tests. 2024-07-26T23:38:09.808 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify_test_cb 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (22192 ms) 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94140948101648 err -107 2024-07-26T23:38:09.809 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (3252 ms) 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94140948145168 err -107 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (775 ms) 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 188978561024 cookie 94140948162912 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (972 ms) 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 188978561024 cookie 94140948162912 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (945 ms) 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-07-26T23:38:09.810 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 188978561024 cookie 94140948202336 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (269 ms) 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 188978561026 cookie 94140948183200 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 188978561026 cookie 94140948187232 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (554 ms) 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 188978561027 cookie 94140948202272 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 188978561028 cookie 94140948202272 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3217 ms) 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-07-26T23:38:09.811 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94140948202272 err -107 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_cb from 4427 notify_id 193273528321 cookie 94140948202272 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5275 ms) 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94140948202272 err -107 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1129 ms) 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (38580 ms total) 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: watch_notify_test_cb 2024-07-26T23:38:09.812 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (39156 ms) 2024-07-26T23:38:09.813 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (39156 ms total) 2024-07-26T23:38:09.813 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: 2024-07-26T23:38:09.813 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-07-26T23:38:09.813 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (107991 ms total) 2024-07-26T23:38:09.813 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-07-26T23:38:10.838 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-07-26T23:38:10.838 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-07-26T23:38:10.838 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-07-26T23:38:10.838 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-07-26T23:38:10.838 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: seed 40937 2024-07-26T23:38:10.838 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (24517 ms) 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (439 ms) 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (2532 ms) 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1791 ms) 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (1145 ms) 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (30424 ms total) 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-07-26T23:38:10.839 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (23014 ms) 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (376 ms) 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (20188 ms) 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (1632 ms) 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (45211 ms total) 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (109037 ms total) 2024-07-26T23:38:10.840 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: Running main() from gmock_main.cc 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [----------] Global test environment set-up. 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStat.Stat (26848 ms) 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-07-26T23:38:11.011 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (1511 ms) 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStat.StatNS (2723 ms) 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (4787 ms) 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1223 ms) 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [----------] 5 tests from LibRadosStat (37092 ms total) 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-07-26T23:38:11.012 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (30210 ms) 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (10294 ms) 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (500 ms) 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (652 ms) 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (41656 ms total) 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [----------] Global test environment tear-down 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (109213 ms total) 2024-07-26T23:38:11.013 INFO:tasks.workunit.client.0.smithi139.stdout: api_stat: [ PASSED ] 9 tests. 2024-07-26T23:38:23.917 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-07-26T23:38:23.917 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-07-26T23:38:23.917 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: notify 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (20395 ms) 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (7854 ms) 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (28249 ms total) 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-07-26T23:38:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: notify 2024-07-26T23:38:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (39038 ms) 2024-07-26T23:38:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-07-26T23:38:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: notify 2024-07-26T23:38:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (13306 ms) 2024-07-26T23:38:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-07-26T23:38:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (3486 ms) 2024-07-26T23:38:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (1004 ms) 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392489952 notify_id 369367187459 notifier_gid 4921 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (382 ms) 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392492880 notify_id 369367187460 notifier_gid 4921 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392492880 notify_id 369367187461 notifier_gid 4921 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (505 ms) 2024-07-26T23:38:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392492880 notify_id 369367187458 notifier_gid 4921 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (475 ms) 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392492880 notify_id 369367187462 notifier_gid 4921 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (1080 ms) 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392649376 notify_id 373662154753 notifier_gid 4921 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (1393 ms) 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-07-26T23:38:23.921 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392662768 notify_id 373662154758 notifier_gid 4921 2024-07-26T23:38:23.922 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (315 ms) 2024-07-26T23:38:23.922 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-07-26T23:38:23.922 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: trying... 2024-07-26T23:38:24.833 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392662768 notif cmd: Running main() from gmock_main.cc 2024-07-26T23:38:24.833 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-07-26T23:38:24.833 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [----------] Global test environment set-up. 2024-07-26T23:38:24.833 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6846 ms) 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (14212 ms) 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (101750 ms) 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [----------] 3 tests from NeoRadosCmd (122808 ms total) 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [----------] Global test environment tear-down 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [==========] 3 tests from 1 test suite ran. (122808 ms total) 2024-07-26T23:38:24.834 INFO:tasks.workunit.client.0.smithi139.stdout: cmd: [ PASSED ] 3 tests. 2024-07-26T23:38:29.780 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:04:37.452 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:29.794 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:04:37.466 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:30.178 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:04:37.849 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:30.599 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:04:38.270 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:30.748 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:38.410 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:30.760 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:04:38.431 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:30.843 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:38.505 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:30.960 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:04:38.632 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:31.073 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:04:38.735 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:38:32.400 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_1_[41304]: starting. 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_1_[41304]: creating pool ceph_test_rados_delete_pools_parallel.smithi139-41260 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_1_[41304]: created object 0... 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_1_[41304]: created object 25... 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_1_[41304]: created object 49... 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_1_[41304]: finishing. 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_1_[41304]: shutting down. 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_2_[41306]: starting. 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_2_[41306]: deleting pool ceph_test_rados_delete_pools_parallel.smithi139-41260 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_2_[41306]: shutting down. 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: ******************************* 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_3_[42985]: starting. 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_3_[42985]: creating pool ceph_test_rados_delete_pools_parallel.smithi139-41260 2024-07-26T23:38:32.401 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_3_[42985]: created object 0... 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_3_[42985]: created object 25... 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_3_[42985]: created object 49... 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_3_[42985]: finishing. 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_3_[42985]: shutting down. 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: ******************************* 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_5_[42987]: starting. 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_5_[42987]: listing objects. 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_5_[42987]: listed object 0... 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_5_[42987]: listed object 25... 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_5_[42987]: saw 50 objects 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_5_[42987]: shutting down. 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: ******************************* 2024-07-26T23:38:32.402 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_4_[42986]: starting. 2024-07-26T23:38:32.406 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_4_[42986]: deleting pool ceph_test_rados_delete_pools_parallel.smithi139-41260 2024-07-26T23:38:32.406 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: process_4_[42986]: shutting down. 2024-07-26T23:38:32.406 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: ******************************* 2024-07-26T23:38:32.406 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: ******************************* 2024-07-26T23:38:32.406 INFO:tasks.workunit.client.0.smithi139.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-07-26T23:38:32.446 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_1_[41352]: starting. 2024-07-26T23:38:32.446 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_1_[41352]: creating pool ceph_test_rados_open_pools_parallel.smithi139-41316 2024-07-26T23:38:32.446 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_1_[41352]: created object 0... 2024-07-26T23:38:32.446 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_1_[41352]: created object 25... 2024-07-26T23:38:32.446 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_1_[41352]: created object 49... 2024-07-26T23:38:32.446 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_1_[41352]: finishing. 2024-07-26T23:38:32.446 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_1_[41352]: shutting down. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_2_[41355]: starting. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_2_[41355]: rados_pool_create. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_2_[41355]: rados_ioctx_create. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_2_[41355]: shutting down. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: ******************************* 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_3_[42988]: starting. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_3_[42988]: creating pool ceph_test_rados_open_pools_parallel.smithi139-41316 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_3_[42988]: created object 0... 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_3_[42988]: created object 25... 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_3_[42988]: created object 49... 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_3_[42988]: finishing. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_3_[42988]: shutting down. 2024-07-26T23:38:32.447 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: ******************************* 2024-07-26T23:38:32.448 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_4_[42989]: starting. 2024-07-26T23:38:32.448 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_4_[42989]: rados_pool_create. 2024-07-26T23:38:32.448 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_4_[42989]: rados_ioctx_create. 2024-07-26T23:38:32.448 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: process_4_[42989]: shutting down. 2024-07-26T23:38:32.448 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: ******************************* 2024-07-26T23:38:32.448 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: ******************************* 2024-07-26T23:38:32.448 INFO:tasks.workunit.client.0.smithi139.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_1_[41311]: starting. 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_1_[41311]: creating pool ceph_test_rados_list_parallel.smithi139-41257 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_1_[41311]: created object 0... 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_1_[41311]: created object 25... 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_1_[41311]: created object 49... 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_1_[41311]: finishing. 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_1_[41311]: shutting down. 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_2_[41312]: starting. 2024-07-26T23:38:32.661 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_2_[41312]: listing objects. 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_2_[41312]: listed object 0... 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_2_[41312]: listed object 25... 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_2_[41312]: saw 50 objects 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_2_[41312]: shutting down. 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_3_[42717]: starting. 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_3_[42717]: creating pool ceph_test_rados_list_parallel.smithi139-41257 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_3_[42717]: created object 0... 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_3_[42717]: created object 25... 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_3_[42717]: created object 49... 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_3_[42717]: finishing. 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_3_[42717]: shutting down. 2024-07-26T23:38:32.662 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_4_[42718]: starting. 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_4_[42718]: listing objects. 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_4_[42718]: listed object 0... 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_4_[42718]: listed object 25... 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_4_[42718]: saw 44 objects 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_4_[42718]: shutting down. 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_5_[42719]: starting. 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_5_[42719]: removed 25 objects... 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_5_[42719]: removed half of the objects 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_5_[42719]: removed 50 objects... 2024-07-26T23:38:32.663 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_5_[42719]: removed 50 objects 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_5_[42719]: shutting down. 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_6_[42781]: starting. 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_6_[42781]: creating pool ceph_test_rados_list_parallel.smithi139-41257 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_6_[42781]: created object 0... 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_6_[42781]: created object 25... 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_6_[42781]: created object 49... 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_6_[42781]: finishing. 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_6_[42781]: shutting down. 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.664 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_7_[42782]: starting. 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_7_[42782]: listing objects. 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_7_[42782]: listed object 0... 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_7_[42782]: listed object 25... 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_7_[42782]: listed object 50... 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_7_[42782]: saw 56 objects 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_7_[42782]: shutting down. 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_8_[42783]: starting. 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_8_[42783]: added 25 objects... 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_8_[42783]: added half of the objects 2024-07-26T23:38:32.665 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_8_[42783]: added 50 objects... 2024-07-26T23:38:32.666 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_8_[42783]: added 50 objects 2024-07-26T23:38:32.666 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_8_[42783]: shutting down. 2024-07-26T23:38:32.666 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:32.666 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:41.018 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ********************** api_lock_pp: Running main() from gmock_main.cc 2024-07-26T23:38:41.018 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-07-26T23:38:41.018 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-07-26T23:38:41.018 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-07-26T23:38:41.018 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: seed 40654 2024-07-26T23:38:41.018 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (28586 ms) 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (183 ms) 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1976 ms) 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1235 ms) 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (281 ms) 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-07-26T23:38:41.019 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (270 ms) 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (171 ms) 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (100 ms) 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (32802 ms total) 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (37749 ms) 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (2544 ms) 2024-07-26T23:38:41.020 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1726 ms) 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1124 ms) 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (11675 ms) 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (1812 ms) 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (290 ms) 2024-07-26T23:38:41.021 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-07-26T23:38:41.022 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (127 ms) 2024-07-26T23:38:41.022 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (57047 ms total) 2024-07-26T23:38:41.022 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: 2024-07-26T23:38:41.022 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-07-26T23:38:41.022 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (139343 ms total) 2024-07-26T23:38:41.022 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-07-26T23:38:45.147 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: Running main() from gmock_main.cc 2024-07-26T23:38:45.759 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-07-26T23:38:45.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [----------] Global test environment set-up. 2024-07-26T23:38:45.760 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-07-26T23:38:45.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-07-26T23:38:45.761 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (16828 ms) 2024-07-26T23:38:45.762 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-07-26T23:38:45.762 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.LockShared (13363 ms) 2024-07-26T23:38:45.763 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-07-26T23:38:45.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1236 ms) 2024-07-26T23:38:45.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-07-26T23:38:45.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1201 ms) 2024-07-26T23:38:45.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (176 ms) 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.Unlock (247 ms) 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (247 ms) 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (94 ms) 2024-07-26T23:38:45.769 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [----------] 8 tests from LibRadosLock (33392 ms total) 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (38613 ms) 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (7993 ms) 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (4762 ms) 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (2724 ms) 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (204 ms) 2024-07-26T23:38:45.770 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (1853 ms) 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (349 ms) 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (135 ms) 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (56633 ms total) 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [----------] Global test environment tear-down 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (143489 ms total) 2024-07-26T23:38:45.771 INFO:tasks.workunit.client.0.smithi139.stdout: api_lock: [ PASSED ] 16 tests. 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout:********* 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_9_[49972]: starting. 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_9_[49972]: creating pool ceph_test_rados_list_parallel.smithi139-41257 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_9_[49972]: created object 0... 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_9_[49972]: created object 25... 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_9_[49972]: created object 49... 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_9_[49972]: finishing. 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_9_[49972]: shutting down. 2024-07-26T23:38:45.772 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: starting. 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: listing objects. 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: listed object 0... 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: listed object 25... 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: listed object 50... 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: listed object 75... 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: saw 100 objects 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_10_[49973]: shutting down. 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.773 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_13_[49976]: starting. 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_13_[49976]: removed 25 objects... 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_13_[49976]: removed half of the objects 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_13_[49976]: removed 50 objects... 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_13_[49976]: removed 50 objects 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_13_[49976]: shutting down. 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_11_[49974]: starting. 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_11_[49974]: added 25 objects... 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_11_[49974]: added half of the objects 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_11_[49974]: added 50 objects... 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_11_[49974]: added 50 objects 2024-07-26T23:38:45.774 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_11_[49974]: shutting down. 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_12_[49975]: starting. 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_12_[49975]: added 25 objects... 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_12_[49975]: added half of the objects 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_12_[49975]: added 50 objects... 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_12_[49975]: added 50 objects 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_12_[49975]: shutting down. 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.775 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_14_[62893]: starting. 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_14_[62893]: creating pool ceph_test_rados_list_parallel.smithi139-41257 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_14_[62893]: created object 0... 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_14_[62893]: created object 25... 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_14_[62893]: created object 49... 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_14_[62893]: finishing. 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_14_[62893]: shutting down. 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: starting. 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: listing objects. 2024-07-26T23:38:45.776 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: listed object 0... 2024-07-26T23:38:52.452 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: listed object api_io: Running main() from gmock_main.cc 2024-07-26T23:38:52.452 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-07-26T23:38:52.452 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [----------] Global test environment set-up. 2024-07-26T23:38:52.452 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-07-26T23:38:52.452 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (27649 ms) 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: no timeout :/ 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: no timeout :/ 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: no timeout :/ 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: no timeout :/ 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: no timeout :/ 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (164 ms) 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (134 ms) 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-07-26T23:38:52.453 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.Checksum (61 ms) 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (3108 ms) 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (740 ms) 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (197 ms) 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (589 ms) 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.TruncTest (96 ms) 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (202 ms) 2024-07-26T23:38:52.454 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (124 ms) 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.RmXattr (387 ms) 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIo.XattrIter (184 ms) 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [----------] 14 tests from LibRadosIo (33635 ms total) 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (36619 ms) 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (4665 ms) 2024-07-26T23:38:52.455 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (11657 ms) 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (1213 ms) 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (1095 ms) 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (285 ms) 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (337 ms) 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (321 ms) 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-07-26T23:38:52.456 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (2626 ms) 2024-07-26T23:38:52.457 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-07-26T23:38:52.457 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (600 ms) 2024-07-26T23:38:52.457 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [----------] 10 tests from LibRadosIoEC (59419 ms total) 2024-07-26T23:38:52.457 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: 2024-07-26T23:38:52.457 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [----------] Global test environment tear-down 2024-07-26T23:38:52.457 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [==========] 24 tests from 2 test suites ran. (150827 ms total) 2024-07-26T23:38:52.457 INFO:tasks.workunit.client.0.smithi139.stdout: api_io: [ PASSED ] 24 tests. 2024-07-26T23:39:01.850 INFO:tasks.workunit.client.0.smithi139.stdout:25... 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: listed object 50... 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: listed object 75... 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: listed object 100... 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: listed object 125... 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: saw 150 objects 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_15_[62894]: shutting down. 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.851 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_16_[62895]: starting. 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_16_[62895]: added 25 objects... 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_16_[62895]: added half of the objects 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_16_[62895]: added 50 objects... 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_16_[62895]: added 50 objects 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: process_16_[62895]: shutting down. 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******************************* 2024-07-26T23:39:01.852 INFO:tasks.workunit.client.0.smithi139.stdout: list_parallel: ******* SUCCESS ********** 2024-07-26T23:39:08.092 INFO:tasks.ceph.mon.c.smithi139.stderr:2024-07-26T23:39:08.067+0000 fff5640 -1 mon.c@2(peon).paxos(paxos updating c 1..322) lease_expire from mon.0 v2:172.21.15.139:3300/0 is 1.106994629s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-07-26T23:39:19.968 INFO:tasks.ceph.mon.c.smithi139.stderr:2024-07-26T23:39:19.957+0000 fff5640 -1 mon.c@2(peon).paxos(paxos updating c 1..326) lease_expire from mon.0 v2:172.21.15.139:3300/0 is 0.030216837s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-07-26T23:39:23.918 INFO:tasks.workunit.client.0.smithi139.stdout:3053bd23e0 2024-07-26T23:39:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: waiting 0x563054484050 2024-07-26T23:39:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: done. 2024-07-26T23:39:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (51937 ms) 2024-07-26T23:39:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (51937 ms total) 2024-07-26T23:39:23.918 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.Exec (9790 ms) 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (25779 ms) 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3568 ms) 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.Applications (11368 ms) 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-07-26T23:39:23.919 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (70148 ms) 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] 8 tests from LibRadosMisc (120653 ms total) 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [----------] Global test environment tear-down 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (182191 ms total) 2024-07-26T23:39:23.920 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc: [ PASSED ] 12 tests. 2024-07-26T23:39:23.957 INFO:tasks.workunit.client.0.smithi139.stdout:y_id 373662154759 notifier_gid 4921 2024-07-26T23:39:23.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: timed out 2024-07-26T23:39:23.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: flushing 2024-07-26T23:39:23.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: flushed 2024-07-26T23:39:23.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3177 ms) 2024-07-26T23:39:23.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-07-26T23:39:23.957 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: trying... 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392662768 notify_id 377957122051 notifier_gid 4921 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: timed out 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: flushing 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: flushed 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3682 ms) 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: List watches 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: notify2 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392662768 notify_id 386547056647 notifier_gid 4921 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: notify2 done 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: watch_check 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: unwatch2 2024-07-26T23:39:23.958 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: flushing 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: done 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (5353 ms) 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: List watches 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: notify2 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: handle_notify cookie 94399392662768 notify_id 390842023944 notifier_gid 4921 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: notify2 done 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: watch_check 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: unwatch2 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: flushing 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: done 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (4158 ms) 2024-07-26T23:39:23.959 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (77354 ms total) 2024-07-26T23:39:23.960 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: 2024-07-26T23:39:23.960 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-07-26T23:39:23.960 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (182104 ms total) 2024-07-26T23:39:23.960 INFO:tasks.workunit.client.0.smithi139.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-07-26T23:39:44.213 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:05:51.884 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.218 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:05:51.889 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.305 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:05:51.976 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.314 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:05:51.985 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.318 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:51.977 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.321 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:51.983 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.458 INFO:tasks.workunit.client.0.smithi139.stdout:ReadOpPP (6250 ms) 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5816 ms) 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (4626 ms) 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (2485 ms) 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (197 ms) 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-07-26T23:39:44.459 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (516 ms) 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (324 ms) 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (298 ms) 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (325 ms) 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (369 ms) 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (2627 ms) 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-07-26T23:39:44.460 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (62994 ms) 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4176 ms) 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (3149 ms) 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3841 ms) 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (3567 ms) 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (137100 ms total) 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [----------] Global test environment tear-down 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (202826 ms total) 2024-07-26T23:39:44.461 INFO:tasks.workunit.client.0.smithi139.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-07-26T23:39:44.462 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:05:52.130 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.496 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:05:52.159 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:39:44.525 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:05:52.196 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.466 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:12.136 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.470 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:12.142 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.472 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:12.144 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.551 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:12.222 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.557 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:12.219 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.578 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:12.249 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.600 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:12.263 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.636 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:12.308 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:04.707 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:12.369 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:24.571 INFO:tasks.workunit.client.0.smithi139.stdout: misc: Running main() from gmock_main.cc 2024-07-26T23:40:24.571 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-07-26T23:40:24.571 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [----------] Global test environment set-up. 2024-07-26T23:40:24.571 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.Version (6874 ms) 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (6884 ms) 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.LongName (72521 ms) 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (37218 ms) 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (49136 ms) 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (18005 ms) 2024-07-26T23:40:24.572 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.Exec (16169 ms) 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (5119 ms) 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (5130 ms) 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.BigObject (14951 ms) 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2629 ms) 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (7811 ms) 2024-07-26T23:40:24.573 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [----------] 12 tests from NeoRadosMisc (242447 ms total) 2024-07-26T23:40:24.574 INFO:tasks.workunit.client.0.smithi139.stdout: misc: 2024-07-26T23:40:24.574 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [----------] Global test environment tear-down 2024-07-26T23:40:24.574 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [==========] 12 tests from 1 test suite ran. (242447 ms total) 2024-07-26T23:40:24.574 INFO:tasks.workunit.client.0.smithi139.stdout: misc: [ PASSED ] 12 tests. 2024-07-26T23:40:29.508 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:37.179 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.559 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:37.231 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.579 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:37.241 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.579 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:37.250 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.580 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:37.251 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.582 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:37.244 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.700 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:37.362 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.704 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:37.375 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:29.737 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:37.408 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.663 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:57.333 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.663 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:57.335 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.670 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:57.341 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.671 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:57.342 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.684 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:57.346 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.687 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:57.349 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.842 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:06:57.513 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.860 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:06:57.521 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:49.886 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:06:57.557 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:40:51.870 INFO:tasks.workunit.client.0.smithi139.stdout: io: Running main() from gmock_main.cc 2024-07-26T23:40:51.870 INFO:tasks.workunit.client.0.smithi139.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-07-26T23:40:51.870 INFO:tasks.workunit.client.0.smithi139.stdout: io: [----------] Global test environment set-up. 2024-07-26T23:40:51.870 INFO:tasks.workunit.client.0.smithi139.stdout: io: [----------] 14 tests from NeoRadosIo 2024-07-26T23:40:51.870 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.Limits (51292 ms) 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (44063 ms) 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.ReadOp (28190 ms) 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.SparseRead (36167 ms) 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.RoundTrip (28907 ms) 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (8095 ms) 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-07-26T23:40:51.871 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (8977 ms) 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (13598 ms) 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (8434 ms) 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.Trunc (11289 ms) 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.Remove (16494 ms) 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (5090 ms) 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-07-26T23:40:51.872 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.RmXattr (4453 ms) 2024-07-26T23:40:51.873 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-07-26T23:40:51.873 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ OK ] NeoRadosIo.GetXattrs (4744 ms) 2024-07-26T23:40:51.873 INFO:tasks.workunit.client.0.smithi139.stdout: io: [----------] 14 tests from NeoRadosIo (269793 ms total) 2024-07-26T23:40:51.873 INFO:tasks.workunit.client.0.smithi139.stdout: io: 2024-07-26T23:40:51.873 INFO:tasks.workunit.client.0.smithi139.stdout: io: [----------] Global test environment tear-down 2024-07-26T23:40:51.873 INFO:tasks.workunit.client.0.smithi139.stdout: io: [==========] 14 tests from 1 test suite ran. (269794 ms total) 2024-07-26T23:40:51.873 INFO:tasks.workunit.client.0.smithi139.stdout: io: [ PASSED ] 14 tests. 2024-07-26T23:40:57.763 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: Running main() from gmock_main.cc 2024-07-26T23:40:57.763 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-07-26T23:40:57.763 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] Global test environment set-up. 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (53208 ms) 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (30707 ms) 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (18864 ms) 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (4370 ms) 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (107149 ms total) 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: 2024-07-26T23:40:57.764 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (46636 ms) 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (10479 ms) 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (10499 ms) 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (67615 ms total) 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-07-26T23:40:57.765 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (10383 ms) 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (7593 ms) 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (6992 ms) 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (4436 ms) 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (29404 ms total) 2024-07-26T23:40:57.766 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (14351 ms) 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (10658 ms) 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (25009 ms total) 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [----------] Global test environment tear-down 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (276043 ms total) 2024-07-26T23:40:57.767 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-07-26T23:41:02.922 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: Running main() from gmock_main.cc 2024-07-26T23:41:02.922 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [----------] Global test environment set-up. 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (32968 ms) 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (53327 ms) 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (37136 ms) 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (35320 ms) 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (30737 ms) 2024-07-26T23:41:02.923 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-07-26T23:41:02.924 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (8275 ms) 2024-07-26T23:41:02.924 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-07-26T23:41:02.924 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (17259 ms) 2024-07-26T23:41:02.924 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-07-26T23:41:02.924 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (7233 ms) 2024-07-26T23:41:02.924 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-07-26T23:41:02.924 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (4265 ms) 2024-07-26T23:41:02.925 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-07-26T23:41:02.925 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (22458 ms) 2024-07-26T23:41:02.925 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-07-26T23:41:02.925 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (14938 ms) 2024-07-26T23:41:02.925 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-07-26T23:41:02.925 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (7864 ms) 2024-07-26T23:41:02.925 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-07-26T23:41:02.926 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3847 ms) 2024-07-26T23:41:02.926 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-07-26T23:41:02.926 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (5187 ms) 2024-07-26T23:41:02.926 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (280814 ms total) 2024-07-26T23:41:02.926 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: 2024-07-26T23:41:02.927 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [----------] Global test environment tear-down 2024-07-26T23:41:02.927 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (280814 ms total) 2024-07-26T23:41:02.927 INFO:tasks.workunit.client.0.smithi139.stdout: read_operations: [ PASSED ] 14 tests. 2024-07-26T23:41:05.255 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: Running main() from gmock_main.cc 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [----------] Global test environment set-up. 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (32907 ms) 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (33956 ms) 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (39586 ms) 2024-07-26T23:41:05.256 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (48330 ms) 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (38982 ms) 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (193761 ms total) 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (14451 ms) 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (13141 ms) 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (17605 ms) 2024-07-26T23:41:05.257 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (7657 ms) 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (11807 ms) 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi139-41671-11. 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: Waiting for snaps to purge. 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (24701 ms) 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (89362 ms total) 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [----------] Global test environment tear-down 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (283124 ms total) 2024-07-26T23:41:05.258 INFO:tasks.workunit.client.0.smithi139.stdout: snapshots: [ PASSED ] 11 tests. 2024-07-26T23:41:09.920 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:17.591 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:09.923 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:17.594 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:09.995 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:17.657 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:09.995 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:17.667 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:10.015 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:17.677 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:10.018 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:17.689 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:10.025 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:17.696 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:10.099 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:17.758 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:10.125 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:17.797 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: seed 40849 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (37115 ms) 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (15072 ms) 2024-07-26T23:41:14.309 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (30976 ms) 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (18933 ms) 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (34777 ms) 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (136873 ms total) 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (12324 ms) 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-07-26T23:41:14.310 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (13527 ms) 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (34525 ms) 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4386 ms) 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (9935 ms) 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5372-g67f28c2a/rpm/el9/BUILD/ceph-19.0.0-5372-g67f28c2a/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-07-26T23:41:14.311 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi139-40849-7 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: waiting for snaps to purge 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19496 ms) 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (94193 ms total) 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (6 ms) 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5275 ms) 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5281 ms total) 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: 2024-07-26T23:41:14.312 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (5042 ms) 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-07-26T23:41:14.313 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: seed 40696 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (527 ms) 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (25112 ms) 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (1197 ms) 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (3765 ms) 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (217 ms) 2024-07-26T23:41:14.314 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (177 ms) 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (89 ms) 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (198 ms) 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (94 ms) 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (229 ms) 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (185 ms) 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (140 ms) 2024-07-26T23:41:14.315 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (258 ms) 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: osd_max_attr_size = 0 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (127014 ms) 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (7132 ms) 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: done waiting, doing copies 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: waiting for final deep scrubs... 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: done waiting 2024-07-26T23:41:14.316 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (113557 ms) 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (2146 ms) 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (316 ms) 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5085 ms) 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-07-26T23:41:14.317 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-07-26T23:41:30.093 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:37.763 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.139 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:37.810 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.159 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:37.822 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.194 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:37.856 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.234 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:37.905 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.290 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:37.961 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.305 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:37.976 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.306 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:37.969 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:30.399 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:38.070 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:32.358 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [--------- api_aio: Running main() from gmock_main.cc 2024-07-26T23:41:32.358 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-07-26T23:41:32.358 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [----------] Global test environment set-up. 2024-07-26T23:41:32.358 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-07-26T23:41:32.358 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-07-26T23:41:32.358 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4544 ms) 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (47214 ms) 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (45274 ms) 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (26897 ms) 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (36276 ms) 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (29773 ms) 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (7186 ms) 2024-07-26T23:41:32.359 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (7807 ms) 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (13674 ms) 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (8369 ms) 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (8122 ms) 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (8808 ms) 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (5528 ms) 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-07-26T23:41:32.360 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3126 ms) 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.Flush (3377 ms) 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (5070 ms) 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (8117 ms) 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (5935 ms) 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (5175 ms) 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4263 ms) 2024-07-26T23:41:32.361 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3279 ms) 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3848 ms) 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4382 ms) 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4305 ms) 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3887 ms) 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (6507 ms) 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [----------] 26 tests from LibRadosAio (310743 ms total) 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: 2024-07-26T23:41:32.362 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-07-26T23:41:38.239 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.Simple api_aio_pp: Running main() from gmock_main.cc 2024-07-26T23:41:38.239 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3981 ms) 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (60724 ms) 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (40002 ms) 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (19213 ms) 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (36278 ms) 2024-07-26T23:41:38.240 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (29742 ms) 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (8295 ms) 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (6751 ms) 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (6271 ms) 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (5300 ms) 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (4242 ms) 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-07-26T23:41:38.241 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (13174 ms) 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3308 ms) 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (9864 ms) 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (12867 ms) 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (4390 ms) 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (7857 ms) 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3915 ms) 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-07-26T23:41:38.242 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (5144 ms) 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (4245 ms) 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3954 ms) 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4181 ms) 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4450 ms) 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (6056 ms) 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (4160 ms) 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-07-26T23:41:38.243 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (4619 ms) 2024-07-26T23:41:38.244 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-07-26T23:41:48.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRaLibRadosSnapshotsECPP.SnapRemovePP (3108 ms) 2024-07-26T23:41:48.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-07-26T23:41:48.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3356 ms) 2024-07-26T23:41:48.137 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (3290 ms) 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (14796 ms total) 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (8633 ms) 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5710 ms) 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3615 ms) 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (17958 ms total) 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: 2024-07-26T23:41:48.138 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-07-26T23:41:48.139 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (326367 ms total) 2024-07-26T23:41:48.139 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-07-26T23:41:48.139 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-07-26T23:41:48.139 INFO:tasks.workunit.client.0.smithi139.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-07-26T23:41:50.030 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:57.692 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.032 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:57.695 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.034 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:57.706 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.041 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:57.712 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.094 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:07:57.756 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.105 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:57.776 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.113 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:57.785 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.170 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:07:57.838 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:41:50.184 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:07:57.856 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:00.515 INFO:tasks.workunit.client.0.smithi139.stdout:-] 22 tests from LibRadosMiscPP (287439 ms total) 2024-07-26T23:42:00.515 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: 2024-07-26T23:42:00.515 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-07-26T23:42:00.515 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-07-26T23:42:00.515 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (6990 ms) 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (6990 ms total) 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (1034 ms) 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (1084 ms) 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (2118 ms total) 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-07-26T23:42:00.516 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (1409 ms) 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (244 ms) 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (1653 ms total) 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (1755 ms) 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (640 ms) 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (2395 ms total) 2024-07-26T23:42:00.517 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (7721 ms) 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (7721 ms total) 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (338775 ms total) 2024-07-26T23:42:00.518 INFO:tasks.workunit.client.0.smithi139.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-07-26T23:42:10.390 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:18.060 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.396 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:18.067 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.474 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:18.136 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.480 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:18.152 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.481 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:18.153 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.493 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:18.164 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.497 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:18.159 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.570 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:18.232 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:10.575 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:18.247 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:11.430 INFO:tasks.workunit.client.0.smithi139.stdout: list: Running main() from gmock_main.cc 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [----------] Global test environment set-up. 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [----------] 3 tests from NeoradosList 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [ OK ] NeoradosList.ListObjects (54950 ms) 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [ OK ] NeoradosList.ListObjectsNS (71514 ms) 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-07-26T23:42:11.431 INFO:tasks.workunit.client.0.smithi139.stdout: list: [ OK ] NeoradosList.ListObjectsMany (222910 ms) 2024-07-26T23:42:11.432 INFO:tasks.workunit.client.0.smithi139.stdout: list: [----------] 3 tests from NeoradosList (349374 ms total) 2024-07-26T23:42:11.432 INFO:tasks.workunit.client.0.smithi139.stdout: list: 2024-07-26T23:42:11.432 INFO:tasks.workunit.client.0.smithi139.stdout: list: [----------] Global test environment tear-down 2024-07-26T23:42:11.432 INFO:tasks.workunit.client.0.smithi139.stdout: list: [==========] 3 tests from 1 test suite ran. (349374 ms total) 2024-07-26T23:42:11.432 INFO:tasks.workunit.client.0.smithi139.stdout: list: [ PASSED ] 3 tests. 2024-07-26T23:42:30.203 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:37.874 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.270 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:37.941 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.273 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:37.941 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.274 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:37.944 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.331 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:38.003 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.332 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:38.003 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.344 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:38.006 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.346 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:38.008 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:30.407 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:38.069 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.430 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:43.101 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.439 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:43.111 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.485 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:43.156 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.514 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:43.185 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.528 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:43.199 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.528 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:43.190 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.535 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:43.198 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.562 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:43.234 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:35.578 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:43.241 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:45.810 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: Running main() from gmock_main.cc 2024-07-26T23:42:45.810 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-07-26T23:42:45.810 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [----------] Global test environment set-up. 2024-07-26T23:42:45.810 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-07-26T23:42:45.810 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-07-26T23:42:45.810 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (59330 ms) 2024-07-26T23:42:45.810 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (77258 ms) 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (247105 ms) 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [----------] 3 tests from NeoradosECList (383693 ms total) 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [----------] Global test environment tear-down 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (383693 ms total) 2024-07-26T23:42:45.811 INFO:tasks.workunit.client.0.smithi139.stdout: ec_list: [ PASSED ] 3 tests. 2024-07-26T23:42:50.768 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:58.439 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.770 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:58.442 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.821 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:58.489 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.825 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:08:58.496 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.842 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:58.513 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.853 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:58.515 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.858 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:58.520 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.909 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:08:58.571 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:50.913 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:08:58.585 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:42:51.470 INFO:tasks.workunit.client.0.smithi139.stdout:i_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=11 expected=11 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=2 expected=2 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:e960b815:::9:head 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-07-26T23:42:51.471 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=9 expected=9 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=12 expected=12 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:863748b0:::15:head 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=15 expected=15 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=0 expected=0 2024-07-26T23:42:51.472 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=14 expected=14 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: : entry=7 expected=7 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (8201 ms) 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (295198 ms) 2024-07-26T23:42:51.473 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (30469 ms) 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] 7 tests from LibRadosList (369862 ms total) 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2198 ms) 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-07-26T23:42:51.474 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset foo1,foo2,foo3 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo1 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo2 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo3 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset foo1,foo4,foo5 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo4 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo5 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo1 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset foo6,foo7 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo7 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: foo6 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-07-26T23:42:51.475 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns1:foo4 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns1:foo5 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns2:foo7 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns2:foo6 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: ns1:foo1 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: :foo1 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: :foo2 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: :foo3 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (7423 ms) 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 1 0 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 10 0 2024-07-26T23:42:51.476 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 13 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 7 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 14 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 0 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 15 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 11 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 5 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 8 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 6 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 3 0 2024-07-26T23:42:59.259 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 4 0 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 12 0 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 9 0 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 2 0 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3472 ms) 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] 3 tests from LibRadosListEC (13093 ms total) 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3070 ms) 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] 1 test from LibRadosListNP (3070 ms total) 2024-07-26T23:42:59.260 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: 2024-07-26T23:42:59.261 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [----------] Global test environment tear-down 2024-07-26T23:42:59.261 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [==========] 11 tests from 3 test suites ran. (397568 ms total) 2024-07-26T23:42:59.261 INFO:tasks.workunit.client.0.smithi139.stdout: api_list: [ PASSED ] 11 tests. 2024-07-26T23:43:03.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-07-26T23:43:03.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-07-26T23:43:03.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-07-26T23:43:03.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-07-26T23:43:03.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-07-26T23:43:03.824 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (15246 ms) 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (39855 ms) 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (19899 ms) 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (10556 ms) 2024-07-26T23:43:03.825 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (34972 ms) 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (39718 ms) 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (241665 ms) 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (401912 ms total) 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (401913 ms total) 2024-07-26T23:43:03.826 INFO:tasks.workunit.client.0.smithi139.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-07-26T23:43:04.790 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: Running main() from gmock_main.cc 2024-07-26T23:43:04.790 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-07-26T23:43:04.790 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [----------] Global test environment set-up. 2024-07-26T23:43:04.790 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-07-26T23:43:04.790 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-07-26T23:43:04.790 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (41435 ms) 2024-07-26T23:43:04.790 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (41513 ms) 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (42205 ms) 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (47538 ms) 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (16745 ms) 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (12339 ms) 2024-07-26T23:43:04.791 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-07-26T23:43:04.792 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (200880 ms) 2024-07-26T23:43:04.792 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (402655 ms total) 2024-07-26T23:43:04.792 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: 2024-07-26T23:43:04.792 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [----------] Global test environment tear-down 2024-07-26T23:43:04.792 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (402655 ms total) 2024-07-26T23:43:04.792 INFO:tasks.workunit.client.0.smithi139.stdout: write_operations: [ PASSED ] 7 tests. 2024-07-26T23:43:15.620 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:09:23.290 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.675 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:09:23.346 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.725 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:23.387 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.727 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:23.390 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.747 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:09:23.418 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.749 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:09:23.421 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.759 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:09:23.430 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.784 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:23.446 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:15.815 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:09:23.486 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:33.851 INFO:tasks.workunit.client.0.smithi139.stdout:Write 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8007 ms) 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8824 ms) 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6997 ms) 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7828 ms) 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-07-26T23:43:33.852 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7405 ms) 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7498 ms) 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8039 ms) 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7972 ms) 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8652 ms) 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7279 ms) 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7421 ms) 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-07-26T23:43:33.853 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6493 ms) 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6947 ms) 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7526 ms) 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6831 ms) 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7772 ms) 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (121491 ms total) 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [----------] Global test environment tear-down 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (432234 ms total) 2024-07-26T23:43:33.854 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio: [ PASSED ] 42 tests. 2024-07-26T23:43:35.765 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:09:43.436 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.770 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:09:43.441 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.813 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:09:43.485 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.823 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:43.486 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.826 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:43.488 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.863 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:09:43.535 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.866 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:09:43.528 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.868 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:09:43.539 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:35.925 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:09:43.597 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:55.918 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:03.588 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:55.985 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:03.656 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:55.986 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:03.648 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:55.988 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:03.651 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:56.028 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:03.690 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:56.047 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:03.718 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:56.063 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:03.735 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:56.067 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:03.738 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:43:56.117 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:03.788 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.125 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:23.796 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.180 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:23.842 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.186 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:23.848 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.190 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:23.861 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.228 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:23.890 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.241 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:23.912 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.260 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:23.931 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.267 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:23.938 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:16.322 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:23.994 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:24.040 INFO:tasks.workunit.client.0.smithi139.stdout:dosAio.RoundTripAppendPP (3632 ms) 2024-07-26T23:44:24.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-07-26T23:44:24.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (4167 ms) 2024-07-26T23:44:24.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-07-26T23:44:24.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3676 ms) 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3088 ms) 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: setting pool EIO 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: max_success 0, min_failed 1 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (6988 ms) 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (5459 ms) 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (339993 ms total) 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-07-26T23:44:24.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3518 ms) 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10410 ms) 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17303 ms) 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3957 ms) 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (35188 ms total) 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3130 ms) 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3130 ms total) 2024-07-26T23:44:24.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (20327 ms) 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6577 ms) 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7051 ms) 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6854 ms) 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (4305 ms) 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6773 ms) 2024-07-26T23:44:24.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-07-26T23:44:24.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7901 ms) 2024-07-26T23:44:24.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-07-26T23:44:24.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7257 ms) 2024-07-26T23:44:24.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-07-26T23:44:24.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7490 ms) 2024-07-26T23:44:24.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-07-26T23:44:24.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6559 ms) 2024-07-26T23:44:24.045 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-07-26T23:44:24.045 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (8279 ms) 2024-07-26T23:44:24.045 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-07-26T23:44:24.045 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6807 ms) 2024-07-26T23:44:24.045 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-07-26T23:44:24.045 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7924 ms) 2024-07-26T23:44:30.795 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:38.466 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.809 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:38.472 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.810 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:38.473 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.812 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:38.484 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.814 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:38.486 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.862 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:38.533 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.896 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:38.558 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.908 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:38.579 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:30.916 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:38.587 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.259 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:43.927 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.315 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:43.977 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.316 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:43.978 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.320 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:43.991 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.341 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:10:44.003 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.343 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:10:44.015 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.379 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:44.051 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.382 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:44.053 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:36.405 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:10:44.076 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.342 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:04.012 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.403 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:04.074 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.411 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:04.074 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.416 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:04.078 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.458 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:04.129 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.468 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:04.130 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.488 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:04.160 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.497 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:04.168 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:44:56.554 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:04.225 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:12.653 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-07-26T23:45:12.653 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (4321 ms) 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7700 ms) 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7988 ms) 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7251 ms) 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7174 ms) 2024-07-26T23:45:12.654 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7130 ms) 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7046 ms) 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (152714 ms total) 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (531025 ms total) 2024-07-26T23:45:12.655 INFO:tasks.workunit.client.0.smithi139.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-07-26T23:45:16.572 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:24.243 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.575 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:24.246 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.621 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:24.292 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.627 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:24.289 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.628 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:24.290 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.644 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:24.315 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.645 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:24.317 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.667 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:24.329 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:16.685 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:24.356 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.516 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:44.187 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.517 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:44.189 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.535 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:11:44.206 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.568 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:44.230 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.569 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:44.231 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.574 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:44.245 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.575 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:44.246 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.586 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:11:44.249 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:45:36.593 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:11:44.264 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.554 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:09.224 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.611 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:09.282 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.652 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:09.323 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.669 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:09.332 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.671 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:09.342 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.671 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:09.333 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.672 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:09.343 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.708 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:09.370 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:01.708 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:09.379 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:21.900 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:29.570 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:21.903 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:29.575 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:21.966 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:29.628 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:22.072 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:29.658 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:22.072 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:29.663 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:22.072 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:29.632 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:22.073 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:29.669 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:22.073 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:29.667 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:22.073 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:29.718 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.778 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:49.448 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.840 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:49.511 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.874 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:49.545 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.894 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:49.557 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.898 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:49.560 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.903 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:49.574 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.904 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:49.576 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.931 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:49.593 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:41.938 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:49.609 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: seed 40815 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (28628 ms) 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-07-26T23:46:42.977 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (72928 ms) 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (3072 ms) 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (104628 ms total) 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (81189 ms) 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (56382 ms) 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (25353 ms) 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-07-26T23:46:42.978 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: my_snaps [3] 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: my_snaps [4,3] 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: my_snaps [5,4,3] 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: promoting some heads 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: promoting from clones for snap 6 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: promoting from clones for snap 5 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: promoting from clones for snap 4 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: promoting from clones for snap 3 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: waiting for scrubs... 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: done waiting 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (150839 ms) 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-07-26T23:46:42.979 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11352 ms) 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10311 ms) 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (9579 ms) 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9246 ms) 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11205 ms) 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8672 ms) 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10561 ms) 2024-07-26T23:46:42.980 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (37357 ms) 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8460 ms) 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10218 ms) 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (14922 ms) 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (10348 ms) 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-07-26T23:46:42.981 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11639 ms) 2024-07-26T23:46:44.107 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:51.778 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.116 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:51.787 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.120 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:51.783 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.122 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:51.794 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.125 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:51.796 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.195 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:51.857 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.213 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:12:51.884 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.229 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:12:51.891 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:44.232 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:12:51.904 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.814 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:04.484 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.878 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:04.549 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.923 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:04.594 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.931 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:04.594 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.935 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:04.598 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.950 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:04.622 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.956 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:04.627 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.963 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:04.626 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:46:56.985 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:04.657 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.014 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:09.686 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.017 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:09.689 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.032 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:09.703 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.071 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:09.734 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.072 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:09.735 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.076 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:09.747 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.081 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:09.752 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.086 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:09.749 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:02.099 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:09.770 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:05.027 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-07-26T23:47:05.027 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: hmm, no HitSet yet 2024-07-26T23:47:05.027 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: ok, hit_set contains 281:602f83fe:::foo:head 2024-07-26T23:47:05.027 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9794 ms) 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg_num = 32 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 0 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 1 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 2 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 3 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 4 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 5 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 6 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 7 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 8 ls 1722037620,0 2024-07-26T23:47:05.028 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 9 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 10 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 11 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 12 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 13 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 14 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 15 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 16 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 17 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 18 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 19 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 20 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 21 ls 1722037620,0 2024-07-26T23:47:05.029 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 22 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 23 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 24 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 25 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 26 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 27 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 28 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 29 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 30 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg 31 ls 1722037620,0 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: pg_num = 32 2024-07-26T23:47:05.030 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6cac518f:::0:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:02547ec2:::1:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f905c69b:::2:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:cfc208b3:::3:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d83876eb:::4:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b29083e3:::5:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c4fdafeb:::6:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:5c6b0b28:::7:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:bd63b0f1:::8:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:e960b815:::9:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:52ea6a34:::10:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:89d3ae78:::11:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:de5d7c5f:::12:head 2024-07-26T23:47:05.031 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:566253c9:::13:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:62a1935d:::14:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:863748b0:::15:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:3958e169:::16:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:4d4dabf9:::17:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:8391935d:::18:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:28883081:::19:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:69259c59:::20:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:4bdb80b7:::21:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a11c5d71:::22:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:271af37b:::23:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:95b121be:::24:head 2024-07-26T23:47:05.032 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:58d1031b:::25:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:0a050783:::26:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c709704c:::27:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:cbe56eaf:::28:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:86b4b162:::29:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:70d89383:::30:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:dd450c7c:::31:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6d5729b1:::32:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c388f3fb:::33:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:56cfea31:::34:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:9dbc1bf7:::35:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:40b74ccd:::36:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:4d5aaf42:::37:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:920f362c:::38:head 2024-07-26T23:47:05.033 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6cc53222:::39:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:9cad833f:::40:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:1ea84d41:::41:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c4480ef6:::42:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a694361e:::43:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d1bd33e9:::44:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:ddc2cd5d:::45:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:2b782207:::46:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:7b187fca:::47:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:90ecdf6f:::48:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a5ed95fe:::49:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:ea0eaa55:::50:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f33ef17b:::51:head 2024-07-26T23:47:05.034 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a0d1b2f6:::52:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:60c5229e:::53:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:edcbc575:::54:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:102cf253:::55:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:efb7fb0b:::56:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:50d0a326:::57:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d4dc5daf:::58:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:3a130462:::59:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:ec87ed71:::60:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d5bc9454:::61:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:3ddfe313:::62:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:7c2816b9:::63:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:47e00e4d:::64:head 2024-07-26T23:47:05.035 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c6410c18:::65:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b48ed237:::66:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:cd63ad31:::67:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b179e92b:::68:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:0d9f741a:::69:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6d3352ae:::70:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c6d5c19e:::71:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:bc4729c3:::72:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:77e930b9:::73:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:0abeecfd:::74:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b7c37e15:::75:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b6378398:::76:head 2024-07-26T23:47:05.036 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:02bd68de:::77:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:cc795d2d:::78:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:630d4fea:::79:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:e0d29ef5:::80:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:fd6f13d2:::81:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:606461d5:::82:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:eadbdc43:::83:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:8761d0bb:::84:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:9ef0186f:::85:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:e0d41294:::86:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:961de695:::87:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:1423148f:::88:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:633a8fa2:::89:head 2024-07-26T23:47:05.037 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a8653809:::90:head 2024-07-26T23:47:05.038 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:3dac8b33:::91:head 2024-07-26T23:47:05.038 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:35aad435:::92:head 2024-07-26T23:47:05.038 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f6dcc343:::93:head 2024-07-26T23:47:05.038 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:dbbdad87:::94:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:1cb48ce0:::95:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:03cd461c:::96:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:17a4ea99:::97:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:9993c9a7:::98:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6394211c:::99:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:94c7ae57:::100:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6fdee5bb:::101:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:9a477fd1:::102:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:eb850916:::103:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:affc56b9:::104:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b42dc814:::105:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f319f8f0:::106:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:9a40b9de:::107:head 2024-07-26T23:47:05.039 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:8b524f28:::108:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:e3de589f:::109:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:90f90a5b:::110:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a7b4f1d7:::111:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:af51766e:::112:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b6f90bd1:::113:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:e0261208:::114:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c9569ef7:::115:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:61bebe50:::116:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:fe93412b:::117:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d3d38bee:::118:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:3100ba0c:::119:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d0560ada:::120:head 2024-07-26T23:47:05.040 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f0ea8b35:::121:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:766f231a:::122:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a07a2582:::123:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:bd7c6b3a:::124:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:fb2ddaff:::125:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:4408e1fe:::126:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:ee1df7a7:::127:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c3002909:::128:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:4f48ffa9:::129:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:edf38733:::130:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c08425c0:::131:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:5f902d98:::132:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:41ea2c93:::133:head 2024-07-26T23:47:05.041 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:813cee13:::134:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:0131818d:::135:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:26ba5a85:::136:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:381b8a5a:::137:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:28797e47:::138:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:bfca7f22:::139:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:36807075:::140:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:80b03975:::141:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:5c15709b:::142:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f39ea15e:::143:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:ea992956:::144:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:48887b1c:::145:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:9f24a9dd:::146:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:987f100b:::147:head 2024-07-26T23:47:05.042 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d2dd3581:::148:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:7fed1808:::149:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c80b70e9:::150:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:85ed90f9:::151:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:36428b24:::152:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d044c34a:::153:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:7c18bf58:::154:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d1c21232:::155:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a7a3c575:::156:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:87da0633:::157:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d5ac3822:::158:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:3f20522d:::159:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6ca26563:::160:head 2024-07-26T23:47:05.043 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:532ce135:::161:head 2024-07-26T23:47:05.044 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:c78863e6:::162:head 2024-07-26T23:47:12.039 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:19.710 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.200 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:19.714 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.200 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:19.728 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.200 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:19.759 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.200 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:19.761 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.200 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:19.776 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.201 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:19.769 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.201 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:19.777 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:12.201 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:19.800 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:21.977 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:29.648 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:21.980 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:29.651 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:22.006 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:29.677 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:22.018 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:29.681 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:22.019 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:29.682 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:22.022 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:29.694 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:22.024 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:29.695 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:22.039 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:29.701 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:22.044 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:29.715 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:26.968 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:34.639 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:26.972 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:34.644 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:26.983 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:34.654 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:27.025 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:34.697 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:27.026 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:34.689 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:27.026 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:34.698 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:27.027 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:34.690 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:27.037 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:34.700 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:27.039 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:34.711 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.221 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:49.892 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.226 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:49.897 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.258 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:13:49.929 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.289 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:49.960 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.290 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:49.952 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.290 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:49.953 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.293 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:49.965 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.316 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:13:49.978 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:42.325 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:13:49.997 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.235 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:04.905 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.237 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:04.908 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.278 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:04.949 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.290 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:04.953 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.291 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:04.954 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.304 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:04.975 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.305 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:04.976 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.320 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:04.982 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:47:57.347 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:05.018 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.156 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:09.827 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.163 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:09.835 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.177 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:09.849 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.215 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:09.877 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.216 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:09.887 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.217 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:09.879 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.220 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:09.891 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.229 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:09.891 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:02.232 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:09.904 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:11.658 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:7462ddf6:::163:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:5d165639:::164:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f43765fc:::165:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b4c720e9:::166:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:e694b040:::167:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:afa38db2:::168:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:77ba9f53:::169:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:87495034:::170:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:7c96bf0e:::171:head 2024-07-26T23:48:11.659 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:dbe346cc:::172:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:e943ec24:::173:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f97a9c0c:::174:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:6f26e74d:::175:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:4f95e106:::176:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:0e6f2f8f:::177:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:05db05f1:::178:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:38a78d66:::179:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:d095610b:::180:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:a1a9d709:::181:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:1e5d39db:::182:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:f7df4fb9:::183:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:03a7f161:::184:head 2024-07-26T23:48:11.660 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:ba70721e:::185:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:28e5662d:::186:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:973d52de:::187:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:4303eb1c:::188:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:b990b48e:::189:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:29b8165b:::190:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:3547f197:::191:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:7e260936:::192:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:1abec7b1:::193:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:10fdda93:::194:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:15817eea:::195:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:770bab57:::196:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:ed9e13e7:::197:head 2024-07-26T23:48:11.661 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:71471a8f:::198:head 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: checking for 282:10fb1d02:::199:head 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21326 ms) 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: first is 1722037642 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,1722037647,1722037648,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,1722037647,1722037648,0 2024-07-26T23:48:11.662 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,1722037647,1722037648,0 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,1722037647,1722037648,1722037650,1722037651,0 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,1722037647,1722037648,1722037650,1722037651,0 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037642,1722037644,1722037645,1722037647,1722037648,1722037650,1722037651,0 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722037645,1722037647,1722037648,1722037650,1722037651,1722037653,1722037654,0 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: first now 1722037645, trimmed 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22394 ms) 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: foo0 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14646 ms) 2024-07-26T23:48:11.663 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-07-26T23:48:12.337 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:20.008 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.410 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:20.071 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.410 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:20.080 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.412 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:20.074 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.429 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:20.092 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.433 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:20.105 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.491 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:20.162 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.499 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:20.171 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:12.531 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:20.203 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.400 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:30.071 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.404 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:30.075 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.427 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:30.098 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.461 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:30.133 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.462 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:30.125 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.463 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:30.134 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.464 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:30.127 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.489 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:30.151 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:22.495 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:30.166 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.215 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:34.886 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.268 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:34.939 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.269 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:34.930 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.271 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:34.934 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.282 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:34.944 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.282 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:34.953 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.323 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:34.995 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.325 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:34.996 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:27.337 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:35.008 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.432 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:50.103 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.436 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:50.107 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.444 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:50.116 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.445 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:50.108 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.505 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:50.176 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.512 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:50.175 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.537 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:50.209 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.546 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:50.209 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:42.554 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:50.226 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.359 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:55.030 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.361 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:55.032 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.375 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:14:55.047 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.412 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:55.075 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.414 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:55.076 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.422 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:55.093 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.423 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:55.095 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.428 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:14:55.090 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:48:47.437 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:14:55.108 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.541 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:10.212 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.544 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:10.215 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.577 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:10.249 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.596 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:10.258 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.600 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:10.262 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.601 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:10.273 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.604 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:10.275 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.628 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:10.291 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:02.636 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:10.306 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.395 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:25.066 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.450 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:25.121 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.478 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:25.149 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.508 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:25.180 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.509 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:25.171 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.511 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:25.182 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.511 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:25.174 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.541 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:25.203 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:17.546 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:25.217 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.552 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:30.223 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.592 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:30.254 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.607 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:30.279 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.608 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:30.280 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.614 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:30.285 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.618 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:30.281 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.624 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:30.295 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.628 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:30.299 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:22.630 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:30.292 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.490 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:40.161 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.494 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:40.166 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.510 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:40.181 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.547 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:40.210 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.549 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:40.211 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.549 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:40.220 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.551 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:40.223 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.563 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:40.225 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:32.567 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:40.238 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.553 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:55.224 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.557 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:55.228 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.583 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:55.254 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.607 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:55.270 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.610 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:55.273 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.613 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:55.285 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.616 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:55.287 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.634 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:55.296 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:47.640 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:55.311 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.402 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:56.064 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.402 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:56.065 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.412 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:15:56.074 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.428 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:56.100 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.429 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:56.101 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.433 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:56.104 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.434 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:56.106 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.439 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:15:56.110 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:49:48.443 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:15:56.114 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.636 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:10.306 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.638 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:10.309 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.664 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:10.335 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.698 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:10.360 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.699 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:10.361 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.715 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:10.385 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.724 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:10.394 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.729 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:10.391 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:02.753 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:10.423 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.718 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:15.389 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.773 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:15.434 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.773 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:15.444 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.774 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:15.435 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.775 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:15.446 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.787 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:15.458 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.795 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:15.465 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.811 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:15.472 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:07.815 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:15.485 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.686 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:25.356 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.689 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:25.359 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.707 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:25.377 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.743 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:25.413 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.743 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:25.405 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.746 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:25.408 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.748 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:25.418 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.762 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:25.423 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:17.769 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:25.439 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.714 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:35.384 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.718 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:35.388 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.731 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:35.401 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.773 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:35.433 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.773 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:35.435 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.780 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:35.450 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.781 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:35.451 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.786 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:35.447 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:27.793 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:35.462 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.873 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:45.542 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.878 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:45.547 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.895 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:45.565 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.927 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:45.588 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.928 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:45.589 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.931 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:45.601 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.932 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:45.602 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.946 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:45.607 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:37.951 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:45.621 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:47.978 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:55.647 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.006 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:55.675 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.031 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:16:55.699 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.048 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:55.708 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.051 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:55.712 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.072 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:16:55.732 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.083 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:55.752 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.094 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:55.763 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:48.120 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:16:55.789 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:52.956 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:00.625 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:52.958 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:00.627 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:52.972 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:00.641 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:53.019 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:00.688 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:53.020 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:00.689 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:53.021 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:00.681 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:53.022 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:00.682 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:53.034 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:00.703 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:50:53.037 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:00.697 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.887 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:15.555 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.890 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:15.559 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.918 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:15.587 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.951 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:15.619 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.954 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:15.614 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.954 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:15.622 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.956 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:15.615 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.979 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:15.647 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:07.979 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:15.639 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.673 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:16.332 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:16.333 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.682 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:16.341 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.691 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:16.360 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.692 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:16.361 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.706 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:16.375 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.711 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:16.380 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.713 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:16.381 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:08.724 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:16.392 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:23.250 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:51:23.238+0000 25291640 -1 snap_mapper.add_oid found existing snaps mapped on 300:847d8473:test-rados-api-smithi139-40815-80::foo:2, removing 2024-07-26T23:51:23.260 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:51:23.248+0000 2c29f640 -1 snap_mapper.add_oid found existing snaps mapped on 300:847d8473:test-rados-api-smithi139-40815-80::foo:2, removing 2024-07-26T23:51:28.079 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:35.746 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.081 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:35.749 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.105 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:35.773 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.135 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:35.794 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.138 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:35.797 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.141 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:35.809 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.144 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:35.812 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.159 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:35.818 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:28.167 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:35.835 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:37.882 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:45.550 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:37.919 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:45.586 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:37.959 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:45.618 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:37.966 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:45.624 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:37.970 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:45.638 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:37.999 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:45.657 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:38.004 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:45.671 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:38.009 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:45.676 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:38.051 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:45.718 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.043 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:55.709 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.060 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:55.728 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.083 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:55.751 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.101 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:55.759 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.102 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:55.760 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.114 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:55.782 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.121 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:55.788 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.123 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:55.781 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.144 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:55.811 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.996 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:56.655 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:48.997 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:56.656 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:49.010 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:17:56.668 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:49.080 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:56.747 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:49.081 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:56.749 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:49.099 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:17:56.762 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:49.176 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:56.843 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:49.177 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:56.844 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:49.189 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:17:56.856 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:51:57.414 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20522 ms) 2024-07-26T23:51:57.414 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23953 ms) 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3301 ms) 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4601 ms) 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4876 ms) 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4421 ms) 2024-07-26T23:51:57.415 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5313 ms) 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40548 ms) 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17099 ms) 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4324 ms) 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-07-26T23:51:57.416 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4269 ms) 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26748 ms) 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16259 ms) 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5462 ms) 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5014 ms) 2024-07-26T23:51:57.417 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: waiting for scrubs... 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: done waiting 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26906 ms) 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11853 ms) 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10889 ms) 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-07-26T23:51:57.418 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:51:57.419 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9915 ms) 2024-07-26T23:52:03.115 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:10.781 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.116 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:10.783 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.146 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:10.812 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.170 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:10.828 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.171 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:10.838 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.171 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:10.829 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.174 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:10.841 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.199 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:10.857 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:03.203 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:10.870 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.141 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:15.808 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.146 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:15.812 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.166 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:15.832 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.211 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:15.868 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.218 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:15.876 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.223 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:15.890 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.227 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:15.893 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.234 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:15.892 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:08.245 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:15.912 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.186 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:25.852 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.188 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:25.855 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.222 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:25.888 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.268 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:25.934 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.276 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:25.943 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.300 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:25.957 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.308 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:25.974 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.316 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:25.973 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:18.345 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:26.002 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.253 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:35.919 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.326 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:35.992 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.327 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:35.984 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.330 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:35.987 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.351 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:36.008 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.355 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:36.021 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.410 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:36.076 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.412 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:36.078 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:28.442 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:36.108 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.213 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:40.878 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.216 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:40.882 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.226 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:40.892 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.266 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:40.923 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.267 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:40.924 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.268 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:40.934 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.272 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:40.938 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.277 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:40.934 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:33.282 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:40.948 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.374 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:56.039 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.378 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:56.043 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.396 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:18:56.061 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.431 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:56.096 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.434 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:56.100 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.445 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:56.101 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.446 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:56.103 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.459 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:18:56.124 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:48.468 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:18:56.125 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.453 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:01.118 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.454 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:01.120 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.469 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:01.134 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.502 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:01.159 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.503 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:01.160 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.509 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:01.175 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.511 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:01.177 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.517 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:01.174 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:52:53.533 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:01.198 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:02.129 INFO:tasks.ceph.osd.3.smithi139.stderr:2024-07-26T23:53:02.089+0000 25291640 -1 snap_mapper.add_oid found existing snaps mapped on 106:11361b24:test-rados-api-smithi139-40815-97::foo:21, removing 2024-07-26T23:53:02.142 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-26T23:53:02.128+0000 27a96640 -1 snap_mapper.add_oid found existing snaps mapped on 106:11361b24:test-rados-api-smithi139-40815-97::foo:21, removing 2024-07-26T23:53:08.544 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:16.209 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.547 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:16.212 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.576 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:16.241 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.596 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:16.253 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.598 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:16.254 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.612 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:16.277 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.613 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:16.278 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.616 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:16.272 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:08.633 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:16.298 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.445 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:31.109 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.449 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:31.113 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.490 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:31.155 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.503 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:31.158 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.504 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:31.160 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.509 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:31.174 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.512 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:31.177 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.534 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:31.189 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:23.544 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:31.208 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.647 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:36.311 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.651 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:36.315 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.675 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:36.339 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.715 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:36.370 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.725 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:36.381 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.734 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:36.398 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.741 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:36.405 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.746 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:36.401 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:28.767 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:36.431 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.732 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:46.396 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.734 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:46.398 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.764 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:46.428 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.788 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:46.443 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.792 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:46.456 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.794 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:46.449 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.795 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:46.459 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.817 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:46.472 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:38.820 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:46.484 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.694 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:55.358 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.755 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:55.419 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.770 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:55.425 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.775 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:55.430 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.786 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:55.450 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.801 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:55.456 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.823 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:55.486 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.824 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:55.488 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:47.852 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:55.515 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.723 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:56.387 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.763 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:56.427 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.764 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:56.428 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.773 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:56.429 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.774 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:19:56.438 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.776 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:56.431 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.784 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:56.448 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.786 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:19:56.441 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:48.796 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:19:56.460 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.693 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:01.356 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.694 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:01.358 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.711 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:01.375 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.748 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:01.402 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.749 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:01.404 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.764 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:01.419 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.766 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:01.429 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.767 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:01.431 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:53:53.783 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:01.446 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.874 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:16.537 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.877 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:16.541 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.929 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:16.592 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.932 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:16.586 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.934 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:16.589 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.952 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:16.616 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.956 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:16.619 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:08.966 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:16.619 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:09.001 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:16.664 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:23.855 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:31.517 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:23.914 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:31.573 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:23.941 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:31.604 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:23.965 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:31.628 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:23.969 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:31.632 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:23.972 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:31.626 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:23.974 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:31.629 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:24.003 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:31.666 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:24.003 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:31.657 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:33.923 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:41.585 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:33.928 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:41.591 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:33.985 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:41.647 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:33.989 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:41.652 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:33.995 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:41.649 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:33.995 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:41.658 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:34.005 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:41.659 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:34.042 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:41.696 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:34.046 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:41.709 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:38.967 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:46.630 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.022 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:46.684 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.033 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:46.687 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.037 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:20:46.700 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.042 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:46.694 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.056 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:20:46.710 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.076 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:46.738 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.077 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:46.740 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:39.100 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:20:46.763 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.933 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:01.595 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.934 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:01.597 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.950 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:01.604 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.954 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:01.608 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.956 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:01.618 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.975 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:01.629 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.989 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:01.652 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:53.992 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:01.655 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.022 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:01.684 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.740 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:02.394 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.741 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:02.395 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.749 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:02.403 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.786 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:02.449 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.787 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:02.450 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.789 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:02.452 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.790 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:02.453 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.796 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:02.458 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:54:54.799 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:02.461 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.093 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:16.755 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.095 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:16.757 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.137 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:16.799 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.150 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:16.803 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.153 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:16.807 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.165 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:16.827 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.166 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:16.828 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.183 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:16.836 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:09.193 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:16.855 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.156 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:21.817 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.160 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:21.822 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.185 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:21.847 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.229 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:21.882 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.233 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:21.895 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.238 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:21.900 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.251 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:21.904 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.266 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:21.926 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:14.273 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:21.926 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.055 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:31.716 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.109 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:31.770 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.135 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:31.796 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.163 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:31.816 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.164 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:31.817 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.165 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:31.826 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.166 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:31.827 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.190 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:31.852 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:24.192 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:31.845 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.165 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:41.826 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.174 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:41.836 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.201 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:41.863 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.225 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:41.887 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.226 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:41.878 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.227 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:41.888 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.230 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:41.883 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.252 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:41.913 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:34.253 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:41.906 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.344 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:52.005 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.347 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:52.008 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.374 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:21:52.036 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.410 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:52.062 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.411 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:52.073 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.412 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:52.064 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.416 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:52.077 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.436 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:21:52.088 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:44.450 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:21:52.111 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.194 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-07-26T23:55:54.194 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:55:54.194 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10483 ms) 2024-07-26T23:55:54.194 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-07-26T23:55:54.194 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:55:54.194 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7731 ms) 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6516 ms) 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26148 ms) 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15140 ms) 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12448 ms) 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-07-26T23:55:54.195 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: 00000010 69 73 |is| 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: 00000012 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12043 ms) 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: require_osd_release = squid 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6075 ms) 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (888651 ms total) 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1463 ms) 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (12206 ms) 2024-07-26T23:55:54.196 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18868 ms) 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (39 ms) 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (32576 ms total) 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8000 ms) 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (10550 ms) 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-07-26T23:55:54.197 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: waiting for scrub... 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: done waiting 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26680 ms) 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10529 ms) 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9920 ms) 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10160 ms) 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12012 ms) 2024-07-26T23:55:54.198 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-07-26T23:55:54.392 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:02.053 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.404 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:02.065 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.417 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.069 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.428 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:02.089 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.434 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:02.095 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.479 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.131 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.517 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:02.169 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.524 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:02.185 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:54.550 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:02.210 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.273 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:06.934 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.325 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:06.986 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.339 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:07.000 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.373 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:07.025 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.376 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:07.028 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.380 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:07.041 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.381 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:07.042 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.390 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:07.042 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:55:59.396 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:07.057 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.518 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:22.177 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.519 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:22.180 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.546 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:22.207 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.574 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:22.225 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.577 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:22.229 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.592 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:22.253 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.593 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:22.254 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.605 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:22.257 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:14.619 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:22.280 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.427 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:37.086 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.485 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:37.136 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.486 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:37.146 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.488 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:37.140 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.523 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:37.174 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.527 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:37.187 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.558 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:37.218 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.562 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:37.223 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:29.603 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:37.264 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.502 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:42.162 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.506 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:42.167 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.516 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:42.167 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.519 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:42.171 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.523 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:42.183 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.535 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:42.186 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.566 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:42.226 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.568 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:42.228 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:34.581 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:42.242 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.662 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:52.321 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.673 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:52.324 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.679 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:52.339 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.728 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:52.388 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.740 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:52.391 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.744 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:52.404 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.768 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:22:52.420 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.785 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:22:52.445 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:44.794 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:22:52.452 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.764 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:02.424 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.769 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:02.429 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.800 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:02.460 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.841 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:02.492 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.847 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:02.495 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.854 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:02.514 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.865 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:02.525 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.869 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:02.520 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:54.892 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:02.552 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.593 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:07.252 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.597 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:07.256 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.611 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:07.270 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.648 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:07.299 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.650 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:07.301 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.652 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:07.312 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.653 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:07.313 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.663 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:07.314 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:56:59.667 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:07.327 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.598 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:22.256 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.654 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:22.314 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.694 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:22.353 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.710 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:22.359 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.711 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:22.360 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.723 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:22.382 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.724 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:22.384 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.735 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:22.386 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:14.750 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:22.410 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.664 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:27.323 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.665 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:27.325 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.679 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:27.339 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.719 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:27.370 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.721 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:27.371 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.723 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:27.383 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.724 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:27.384 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.732 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:27.382 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:19.735 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:27.394 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.804 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:37.463 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.861 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:37.511 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.862 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:37.522 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.865 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:37.524 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.865 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:37.516 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.868 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:37.527 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.886 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:37.536 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.896 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:37.555 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:29.897 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:37.557 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:39.961 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:47.620 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:39.964 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:47.623 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:39.993 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:47.652 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:40.022 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:47.672 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:40.027 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:47.677 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:40.040 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:47.699 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:40.046 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:47.705 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:40.051 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:47.701 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:40.080 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:47.738 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:44.880 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:52.538 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:44.954 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:52.613 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:44.973 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:23:52.632 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:45.034 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:52.683 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:45.035 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:52.694 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:45.035 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:52.685 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:45.042 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:52.701 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:45.058 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:23:52.708 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:45.067 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:23:52.726 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.860 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:07.518 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.864 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:07.522 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.883 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:07.542 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.916 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:07.575 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.920 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:07.578 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.920 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:07.570 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.921 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:07.571 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.940 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:07.590 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:57:59.956 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:07.614 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:04.974 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:12.632 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:04.980 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:12.639 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:05.002 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:12.660 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:05.054 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:12.710 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:05.058 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:12.717 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:05.059 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:12.708 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:05.065 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:12.715 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:05.077 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:12.736 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:05.083 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:12.733 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.430 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:16.089 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.435 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:16.094 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.454 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:16.112 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.500 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:16.149 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.501 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:16.151 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.510 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:16.169 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.513 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:16.171 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.519 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:16.168 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:08.533 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:16.191 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.124 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:27.781 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.131 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:27.789 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.154 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:27.812 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.188 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:27.846 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.189 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:27.848 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.190 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:27.840 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.192 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:27.841 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.214 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:27.872 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:20.216 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:27.865 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.004 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:42.661 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.058 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:42.716 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.085 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:42.743 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.115 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:42.773 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.115 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:42.765 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.116 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:42.775 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.117 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:42.766 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.142 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:42.791 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:35.143 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:42.800 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.098 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:47.756 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.113 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:47.762 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.122 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:47.780 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.164 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:47.822 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.186 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:47.844 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.187 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:47.837 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.198 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:47.856 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.204 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:47.862 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:40.258 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:47.907 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.339 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:57.997 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.400 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:58.057 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.421 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:58.070 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.424 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:58.073 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.443 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:58.101 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.463 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:58.112 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.501 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:58.159 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.502 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:58.160 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.543 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:58.201 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:50.931 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8726 ms) 2024-07-26T23:58:50.931 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-07-26T23:58:50.931 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (15592 ms) 2024-07-26T23:58:50.931 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-07-26T23:58:50.931 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9836 ms) 2024-07-26T23:58:50.931 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14217 ms) 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9765 ms) 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9916 ms) 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: hmm, no HitSet yet 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10122 ms) 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,0 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: first is 1722038231 2024-07-26T23:58:50.932 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,1722038237,1722038238,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,1722038237,1722038238,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,1722038237,1722038238,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,1722038237,1722038238,1722038240,1722038241,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,1722038237,1722038238,1722038240,1722038241,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038231,1722038234,1722038235,1722038237,1722038238,1722038240,1722038241,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: got ls 1722038235,1722038237,1722038238,1722038240,1722038241,1722038243,1722038244,0 2024-07-26T23:58:50.933 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: first now 1722038235, trimmed 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22139 ms) 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: foo0 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (17471 ms) 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20449 ms) 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25238 ms) 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5668 ms) 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-07-26T23:58:50.934 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4375 ms) 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4439 ms) 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3876 ms) 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (269680 ms total) 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1349145 ms total) 2024-07-26T23:58:50.935 INFO:tasks.workunit.client.0.smithi139.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40487 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40487 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40663 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40663 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41176 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41176 2024-07-26T23:58:50.942 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40958 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40958 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40831 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40831 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40558 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40558 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40922 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40922 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41291 2024-07-26T23:58:50.943 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41291 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41328 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41328 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40580 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40580 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41204 2024-07-26T23:58:50.944 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41204 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40460 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40460 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40525 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40525 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41101 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41101 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40451 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40451 2024-07-26T23:58:50.945 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40471 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40471 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41432 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41432 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40725 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40725 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41491 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41491 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.946 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40884 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40884 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41265 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41265 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41559 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41559 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40794 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40794 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.947 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41151 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41151 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41388 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41388 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41234 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41234 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.948 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41469 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41469 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41079 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41079 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40623 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40623 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41546 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41546 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40992 2024-07-26T23:58:50.949 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40992 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40444 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40444 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40504 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40504 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40940 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40940 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=40542 2024-07-26T23:58:50.950 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 40542 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41008 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41008 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41040 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41040 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41341 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41341 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ for t in "${!pids[@]}" 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ pid=41601 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ wait 41601 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ exit 0 2024-07-26T23:58:50.951 INFO:tasks.workunit.client.0.smithi139.stderr:+ cleanup 2024-07-26T23:58:50.952 INFO:tasks.workunit.client.0.smithi139.stderr:+ pkill -P 40438 2024-07-26T23:58:51.030 INFO:tasks.workunit.client.0.smithi139.stderr:+ true 2024-07-26T23:58:51.031 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-07-26T23:58:51.031 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-07-26T23:58:51.058 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-07-26T23:58:51.059 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-07-26T23:58:51.301 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:58.949 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.301 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:58.950 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.311 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:24:58.960 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.414 DEBUG:teuthology.parallel:result is None 2024-07-26T23:58:51.414 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-07-26T23:58:51.436 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:59.094 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.438 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:59.096 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.440 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-07-26T23:58:51.440 DEBUG:teuthology.orchestra.run.smithi139:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-07-26T23:58:51.452 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:24:59.110 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.475 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-07-26T23:58:51.475 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-07-26T23:58:51.486 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T23:58:51.486 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T23:58:51.487 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:59.135 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.487 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:59.136 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.488 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:24:59.146 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.572+0000 7f07eb74d640 10 monclient: get_monmap_and_config 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.572+0000 7f07eb74d640 10 monclient: build_initial_monmap 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.572+0000 7f07eb74d640 10 monclient: monmap: 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:51.573190+0000 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:51.573190+0000 2024-07-26T23:58:51.587 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:51.588 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:51.588 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:51.588 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:51.588 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:51.588 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:51.588 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:51.588 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:51.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.572+0000 7f07eb74d640 1 Processor -- start 2024-07-26T23:58:51.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.572+0000 7f07eb74d640 1 -- start start 2024-07-26T23:58:51.589 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.572+0000 7f07eb74d640 10 monclient: init 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07e406e720 addr v1:172.21.15.139:6789/0 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f07e4143d00 0x7f07e413a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient(hunting): picked mon.noname-a con 0x7f07e4143d00 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e4144cd0 0x7f07e4145100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient(hunting): picked mon.noname-b con 0x7f07e4144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e40ac810 con 0x7f07e406e720 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f07e406eb00 con 0x7f07e4143d00 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:51.590 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f07e4144710 con 0x7f07e4144cd0 2024-07-26T23:58:51.591 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:51.591 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.573+0000 7f07eb74d640 10 monclient(hunting): authenticate will time out at 2764.609617s 2024-07-26T23:58:51.591 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.575+0000 7f07e8ce8640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f07e4143d00 0x7f07e413a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:51.591 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.576+0000 7f07e9cea640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e4144cd0 0x7f07e4145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:51.591 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.576+0000 7f07e8ce8640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f07e4143d00 0x7f07e413a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36224/0 (socket says 172.21.15.139:36224) 2024-07-26T23:58:51.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.576+0000 7f07e8ce8640 1 -- 172.21.15.139:0/2140191325 learned_addr learned my addr 172.21.15.139:0/2140191325 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:51.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.576+0000 7f07e8ce8640 10 monclient(hunting): get_auth_request con 0x7f07e4143d00 auth_method 0 2024-07-26T23:58:51.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.576+0000 7f07e8ce8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:51.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.576+0000 7f07e8ce8640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:51.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.576+0000 7f07e8ce8640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:51.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.577+0000 7f07e8ce8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:51.592 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.577+0000 7f07e8ce8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:51.593 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.578+0000 7f07e8ce8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:51.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.579+0000 7f07e9cea640 10 monclient(hunting): get_auth_request con 0x7f07e4144cd0 auth_method 0 2024-07-26T23:58:51.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.579+0000 7f07e9cea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:51.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.579+0000 7f07e9cea640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:51.594 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.579+0000 7f07e9cea640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:51.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 10 monclient(hunting): handle_auth_done global_id 15519 payload 274 2024-07-26T23:58:51.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 1 -- 172.21.15.139:0/2140191325 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e4144cd0 msgr2=0x7f07e4145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:51.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 1 --2- 172.21.15.139:0/2140191325 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e4144cd0 0x7f07e4145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:51.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 1 -- 172.21.15.139:0/2140191325 >> v1:172.21.15.139:6789/0 conn(0x7f07e406e720 legacy=0x7f07e4139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:51.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 10 monclient: _finish_hunting 0 2024-07-26T23:58:51.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 1 monclient: found mon.noname-a 2024-07-26T23:58:51.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:51.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:51.599 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.580+0000 7f07e8ce8640 1 -- 172.21.15.139:0/2140191325 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e40acaf0 con 0x7f07e4143d00 2024-07-26T23:58:51.600 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.585+0000 7f07e8ce8640 1 --2- 172.21.15.139:0/2140191325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 0x7f07e413a2e0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009870 tx=0x7f07d402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508418c05edb0255 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:51.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 1 -- 172.21.15.139:0/2140191325 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4034070 con 0x7f07e4143d00 2024-07-26T23:58:51.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:51.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:51.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: dump: 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: _finish_auth 0 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: _check_auth_tickets 2024-07-26T23:58:51.627 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 1 -- 172.21.15.139:0/2140191325 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d4030840 con 0x7f07e4143d00 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 1 -- 172.21.15.139:0/2140191325 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4030c10 con 0x7f07e4143d00 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07dbfff640 10 monclient: dump: 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:51.628 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:51.629 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:51.629 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:51.629 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:51.629 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07eb74d640 5 monclient: authenticate success, global_id 15519 2024-07-26T23:58:51.629 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07eb74d640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:51.629 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.611+0000 7f07eb74d640 4 set_mon_vals no callback set 2024-07-26T23:58:51.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 10 monclient: shutdown 2024-07-26T23:58:51.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:51.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 -- 172.21.15.139:0/2140191325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 msgr2=0x7f07e413a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:51.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2140191325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 0x7f07e413a2e0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009870 tx=0x7f07d402b980 comp rx=0 tx=0).stop 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 -- 172.21.15.139:0/2140191325 shutdown_connections 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2140191325 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e4144cd0 0x7f07e4145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2140191325 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 0x7f07e413a2e0 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 -- 172.21.15.139:0/2140191325 >> 172.21.15.139:0/2140191325 conn(0x7f07e40b9c10 msgr2=0x7f07e40ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 -- 172.21.15.139:0/2140191325 shutdown_connections 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.612+0000 7f07eb74d640 1 -- 172.21.15.139:0/2140191325 wait complete. 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.613+0000 7f07eb74d640 10 monclient: build_initial_monmap 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.613+0000 7f07eb74d640 10 monclient: monmap: 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:51.632 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.614+0000 7f07eb74d640 1 librados: starting msgr at 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.614+0000 7f07eb74d640 1 librados: starting objecter 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.614+0000 7f07eb74d640 1 Processor -- start 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.614+0000 7f07eb74d640 1 -- start start 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.614+0000 7f07eb74d640 1 librados: setting wanted keys 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.614+0000 7f07eb74d640 1 librados: calling monclient init 2024-07-26T23:58:51.633 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.614+0000 7f07eb74d640 10 monclient: init 2024-07-26T23:58:51.634 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:51.634 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:58:51.634 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e406e720 0x7f07e4148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:51.634 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): picked mon.noname-c con 0x7f07e406e720 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:51.634 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 0x7f07e41358f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:51.634 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): picked mon.noname-a con 0x7f07e4143d00 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:51.634 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07e4144cd0 0x7f07e4135e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): picked mon.noname-b con 0x7f07e4144cd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4145640 con 0x7f07e4143d00 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4141390 con 0x7f07e4144cd0 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f07e41457c0 con 0x7f07e406e720 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:51.635 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.615+0000 7f07eb74d640 10 monclient(hunting): authenticate will time out at 2764.651462s 2024-07-26T23:58:51.636 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.619+0000 7f07e9cea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07e4144cd0 0x7f07e4135e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:51.636 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.619+0000 7f07e9cea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07e4144cd0 0x7f07e4135e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:46800/0 (socket says 172.21.15.139:46800) 2024-07-26T23:58:51.636 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.619+0000 7f07e9cea640 1 -- 172.21.15.139:0/2377619937 learned_addr learned my addr 172.21.15.139:0/2377619937 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:51.636 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.619+0000 7f07e9cea640 10 monclient(hunting): get_auth_request con 0x7f07e4144cd0 auth_method 0 2024-07-26T23:58:51.636 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.619+0000 7f07e9cea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:51.636 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.619+0000 7f07e9cea640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:51.636 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.619+0000 7f07e9cea640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:51.637 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.620+0000 7f07e94e9640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e406e720 0x7f07e4148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:51.637 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.622+0000 7f07e9cea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:51.637 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.622+0000 7f07e9cea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:51.637 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.622+0000 7f07e9cea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:51.637 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.622+0000 7f07e8ce8640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 0x7f07e41358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:51.640 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 10 monclient(hunting): handle_auth_done global_id 20251 payload 824 2024-07-26T23:58:51.640 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 1 -- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e406e720 msgr2=0x7f07e4148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:51.640 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e406e720 0x7f07e4148190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:51.640 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 1 -- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 msgr2=0x7f07e41358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:51.641 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 0x7f07e41358f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:51.641 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 10 monclient: _finish_hunting 0 2024-07-26T23:58:51.641 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 1 monclient: found mon.noname-b 2024-07-26T23:58:51.641 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:51.641 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:51.641 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.625+0000 7f07e9cea640 1 -- 172.21.15.139:0/2377619937 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e41493c0 con 0x7f07e4144cd0 2024-07-26T23:58:51.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.631+0000 7f07e9cea640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07e4144cd0 0x7f07e4135e30 secure :-1 s=READY pgs=4924 cs=0 l=1 rev1=1 crypto rx=0x7f07dc007e60 tx=0x7f07dc00e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:51.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.631+0000 7f07d9ffb640 10 client.?.objecter ms_handle_connect 0x7f07e4144cd0 2024-07-26T23:58:51.646 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.631+0000 7f07d9ffb640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:51.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 1 -- 172.21.15.139:0/2377619937 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc014420 con 0x7f07e4144cd0 2024-07-26T23:58:51.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:51.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:51.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 10 monclient: dump: 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 10 monclient: _finish_auth 0 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:58:51.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 1 -- 172.21.15.139:0/2377619937 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07dc019070 con 0x7f07e4144cd0 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07d9ffb640 1 -- 172.21.15.139:0/2377619937 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc014820 con 0x7f07e4144cd0 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07eb74d640 5 monclient: authenticate success, global_id 20251 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07eb74d640 10 monclient: _renew_subs 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07eb74d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e413fd90 con 0x7f07e4144cd0 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.655+0000 7f07db7fe640 4 set_mon_vals no callback set 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07d9ffb640 10 monclient: dump: 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:51.673 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07eb74d640 10 client.20251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07eb74d640 10 monclient: _renew_subs 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07eb74d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:51.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e4140020 con 0x7f07e4144cd0 2024-07-26T23:58:51.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.656+0000 7f07eb74d640 1 librados: init done 2024-07-26T23:58:51.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.657+0000 7f07eb74d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:51.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.657+0000 7f07eb74d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:51.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.657+0000 7f07eb74d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:51.675 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.657+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e4145100 con 0x7f07e4144cd0 2024-07-26T23:58:51.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.903+0000 7f07d9ffb640 1 -- 172.21.15.139:0/2377619937 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f07dc0149c0 con 0x7f07e4144cd0 2024-07-26T23:58:51.918 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.903+0000 7f07d9ffb640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f07c80789f0 0x7f07c807aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:51.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.904+0000 7f07e94e9640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f07c80789f0 0x7f07c807aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:51.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.904+0000 7f07e94e9640 10 monclient: get_auth_request con 0x7f07c80789f0 auth_method 0 2024-07-26T23:58:51.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.904+0000 7f07e94e9640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f07c80789f0 0x7f07c807aeb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f07e00015d0 tx=0x7f07e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:51.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.904+0000 7f07d9ffb640 10 client.20251.objecter ms_handle_connect 0x7f07c80789f0 2024-07-26T23:58:51.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.906+0000 7f07d9ffb640 1 -- 172.21.15.139:0/2377619937 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(969..969 src has 501..969) ==== 6365+0+0 (secure 0 0 0) 0x7f07dc05ecc0 con 0x7f07e4144cd0 2024-07-26T23:58:51.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.906+0000 7f07d9ffb640 10 client.20251.objecter ms_dispatch 0x7f07e4000b90 osd_map(969..969 src has 501..969) 2024-07-26T23:58:51.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.906+0000 7f07d9ffb640 3 client.20251.objecter handle_osd_map got epochs [969,969] > 0 2024-07-26T23:58:51.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.906+0000 7f07d9ffb640 3 client.20251.objecter handle_osd_map decoding full epoch 969 2024-07-26T23:58:51.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.906+0000 7f07d9ffb640 20 client.20251.objecter dump_active .. 0 homeless 2024-07-26T23:58:51.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.919+0000 7f07d9ffb640 1 -- 172.21.15.139:0/2377619937 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f07dc063130 con 0x7f07e4144cd0 2024-07-26T23:58:51.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.919+0000 7f07d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:51.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:51.919+0000 7f07d9ffb640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:52.164 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.149+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f07e4145d80 con 0x7f07c80789f0 2024-07-26T23:58:52.168 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.153+0000 7f07d9ffb640 1 -- 172.21.15.139:0/2377619937 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116459 (secure 0 0 0) 0x7f07e4145d80 con 0x7f07c80789f0 2024-07-26T23:58:52.169 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:58:52.170 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-07-26T23:58:52.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.155+0000 7f07eb74d640 10 librados: watch_flush enter 2024-07-26T23:58:52.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.155+0000 7f07eb74d640 10 librados: watch_flush exit 2024-07-26T23:58:52.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.155+0000 7f07eb74d640 20 client.20251.objecter shutdown clearing up homeless session... 2024-07-26T23:58:52.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.155+0000 7f07eb74d640 10 client.20251.objecter successfully canceled tick 2024-07-26T23:58:52.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f07c80789f0 msgr2=0x7f07c807aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f07c80789f0 0x7f07c807aeb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f07e00015d0 tx=0x7f07e0009040 comp rx=0 tx=0).stop 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 10 monclient: shutdown 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07e4144cd0 msgr2=0x7f07e4135e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07e4144cd0 0x7f07e4135e30 secure :-1 s=READY pgs=4924 cs=0 l=1 rev1=1 crypto rx=0x7f07dc007e60 tx=0x7f07dc00e4b0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 shutdown_connections 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f07c80789f0 0x7f07c807aeb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07e4144cd0 0x7f07e4135e30 unknown :-1 s=CLOSED pgs=4924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.173 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e4143d00 0x7f07e41358f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 --2- 172.21.15.139:0/2377619937 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f07e406e720 0x7f07e4148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 >> 172.21.15.139:0/2377619937 conn(0x7f07e40b9c10 msgr2=0x7f07e4146b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:52.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 shutdown_connections 2024-07-26T23:58:52.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 -- 172.21.15.139:0/2377619937 wait complete. 2024-07-26T23:58:52.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.156+0000 7f07eb74d640 1 librados: shutdown 2024-07-26T23:58:52.182 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":1687,"stamp":"2024-07-26T23:58:50.797604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":52,"num_read_kb":43,"num_write":66,"num_write_kb":593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":713040,"kb_used_data":120628,"kb_used_omap":12,"kb_used_meta":592179,"kb_avail":754261680,"statfs":{"total":773094113280,"available":772363960320,"internally_reserved":0,"allocated":123523072,"data_stored":77645195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":606391856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":439,"apply_latency_ms":439,"commit_latency_ns":439000000,"apply_latency_ns":439000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.582548"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1850,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.132348+0000","last_change":"2024-07-26T23:36:37.015743+0000","last_active":"2024-07-26T23:58:46.132348+0000","last_peered":"2024-07-26T23:58:46.132348+0000","last_clean":"2024-07-26T23:58:46.132348+0000","last_became_active":"2024-07-26T23:36:37.010177+0000","last_became_peered":"2024-07-26T23:36:37.010177+0000","last_unstale":"2024-07-26T23:58:46.132348+0000","last_undegraded":"2024-07-26T23:58:46.132348+0000","last_fullsized":"2024-07-26T23:58:46.132348+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:54:54.240713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1858,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.292219+0000","last_change":"2024-07-26T23:36:32.386250+0000","last_active":"2024-07-26T23:58:45.292219+0000","last_peered":"2024-07-26T23:58:45.292219+0000","last_clean":"2024-07-26T23:58:45.292219+0000","last_became_active":"2024-07-26T23:36:32.380680+0000","last_became_peered":"2024-07-26T23:36:32.380680+0000","last_unstale":"2024-07-26T23:58:45.292219+0000","last_undegraded":"2024-07-26T23:58:45.292219+0000","last_fullsized":"2024-07-26T23:58:45.292219+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:19:53.312577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1846,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.439029+0000","last_change":"2024-07-26T23:36:32.478927+0000","last_active":"2024-07-26T23:58:49.439029+0000","last_peered":"2024-07-26T23:58:49.439029+0000","last_clean":"2024-07-26T23:58:49.439029+0000","last_became_active":"2024-07-26T23:36:32.473374+0000","last_became_peered":"2024-07-26T23:36:32.473374+0000","last_unstale":"2024-07-26T23:58:49.439029+0000","last_undegraded":"2024-07-26T23:58:49.439029+0000","last_fullsized":"2024-07-26T23:58:49.439029+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:13:31.972041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1842,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.179188+0000","last_change":"2024-07-26T23:36:45.735883+0000","last_active":"2024-07-26T23:58:45.179188+0000","last_peered":"2024-07-26T23:58:45.179188+0000","last_clean":"2024-07-26T23:58:45.179188+0000","last_became_active":"2024-07-26T23:36:45.718574+0000","last_became_peered":"2024-07-26T23:36:45.718574+0000","last_unstale":"2024-07-26T23:58:45.179188+0000","last_undegraded":"2024-07-26T23:58:45.179188+0000","last_fullsized":"2024-07-26T23:58:45.179188+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:22:53.643566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1844,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.080302+0000","last_change":"2024-07-26T23:36:33.937849+0000","last_active":"2024-07-26T23:58:45.080302+0000","last_peered":"2024-07-26T23:58:45.080302+0000","last_clean":"2024-07-26T23:58:45.080302+0000","last_became_active":"2024-07-26T23:36:33.915707+0000","last_became_peered":"2024-07-26T23:36:33.915707+0000","last_unstale":"2024-07-26T23:58:45.080302+0000","last_undegraded":"2024-07-26T23:58:45.080302+0000","last_fullsized":"2024-07-26T23:58:45.080302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.705473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"44'7","reported_seq":1854,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.005889+0000","last_change":"2024-07-26T23:36:30.524945+0000","last_active":"2024-07-26T23:58:48.005889+0000","last_peered":"2024-07-26T23:58:48.005889+0000","last_clean":"2024-07-26T23:58:48.005889+0000","last_became_active":"2024-07-26T23:36:30.507651+0000","last_became_peered":"2024-07-26T23:36:30.507651+0000","last_unstale":"2024-07-26T23:58:48.005889+0000","last_undegraded":"2024-07-26T23:58:48.005889+0000","last_fullsized":"2024-07-26T23:58:48.005889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:36:56.132284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1845,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.428542+0000","last_change":"2024-07-26T23:36:29.482671+0000","last_active":"2024-07-26T23:58:49.428542+0000","last_peered":"2024-07-26T23:58:49.428542+0000","last_clean":"2024-07-26T23:58:49.428542+0000","last_became_active":"2024-07-26T23:36:29.468516+0000","last_became_peered":"2024-07-26T23:36:29.468516+0000","last_unstale":"2024-07-26T23:58:49.428542+0000","last_undegraded":"2024-07-26T23:58:49.428542+0000","last_fullsized":"2024-07-26T23:58:49.428542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:10:56.313880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1871,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.705157+0000","last_change":"2024-07-26T23:36:50.102068+0000","last_active":"2024-07-26T23:58:49.705157+0000","last_peered":"2024-07-26T23:58:49.705157+0000","last_clean":"2024-07-26T23:58:49.705157+0000","last_became_active":"2024-07-26T23:36:50.085394+0000","last_became_peered":"2024-07-26T23:36:50.085394+0000","last_unstale":"2024-07-26T23:58:49.705157+0000","last_undegraded":"2024-07-26T23:58:49.705157+0000","last_fullsized":"2024-07-26T23:58:49.705157+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:03:21.264586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1845,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.436927+0000","last_change":"2024-07-26T23:36:32.322928+0000","last_active":"2024-07-26T23:58:49.436927+0000","last_peered":"2024-07-26T23:58:49.436927+0000","last_clean":"2024-07-26T23:58:49.436927+0000","last_became_active":"2024-07-26T23:36:32.315924+0000","last_became_peered":"2024-07-26T23:36:32.315924+0000","last_unstale":"2024-07-26T23:58:49.436927+0000","last_undegraded":"2024-07-26T23:58:49.436927+0000","last_fullsized":"2024-07-26T23:58:49.436927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:29:47.697657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1843,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.121254+0000","last_change":"2024-07-26T23:36:33.078206+0000","last_active":"2024-07-26T23:58:45.121254+0000","last_peered":"2024-07-26T23:58:45.121254+0000","last_clean":"2024-07-26T23:58:45.121254+0000","last_became_active":"2024-07-26T23:36:33.069066+0000","last_became_peered":"2024-07-26T23:36:33.069066+0000","last_unstale":"2024-07-26T23:58:45.121254+0000","last_undegraded":"2024-07-26T23:58:45.121254+0000","last_fullsized":"2024-07-26T23:58:45.121254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:58:53.548672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1860,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.299327+0000","last_change":"2024-07-26T23:36:32.994273+0000","last_active":"2024-07-26T23:58:45.299327+0000","last_peered":"2024-07-26T23:58:45.299327+0000","last_clean":"2024-07-26T23:58:45.299327+0000","last_became_active":"2024-07-26T23:36:32.984857+0000","last_became_peered":"2024-07-26T23:36:32.984857+0000","last_unstale":"2024-07-26T23:58:45.299327+0000","last_undegraded":"2024-07-26T23:58:45.299327+0000","last_fullsized":"2024-07-26T23:58:45.299327+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:54:04.890736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.141761+0000","last_change":"2024-07-26T23:36:30.566094+0000","last_active":"2024-07-26T23:58:48.141761+0000","last_peered":"2024-07-26T23:58:48.141761+0000","last_clean":"2024-07-26T23:58:48.141761+0000","last_became_active":"2024-07-26T23:36:30.560131+0000","last_became_peered":"2024-07-26T23:36:30.560131+0000","last_unstale":"2024-07-26T23:58:48.141761+0000","last_undegraded":"2024-07-26T23:58:48.141761+0000","last_fullsized":"2024-07-26T23:58:48.141761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:58:36.316935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1842,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.149181+0000","last_change":"2024-07-26T23:36:44.212854+0000","last_active":"2024-07-26T23:58:45.149181+0000","last_peered":"2024-07-26T23:58:45.149181+0000","last_clean":"2024-07-26T23:58:45.149181+0000","last_became_active":"2024-07-26T23:36:44.149525+0000","last_became_peered":"2024-07-26T23:36:44.149525+0000","last_unstale":"2024-07-26T23:58:45.149181+0000","last_undegraded":"2024-07-26T23:58:45.149181+0000","last_fullsized":"2024-07-26T23:58:45.149181+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:09:35.012505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'32","reported_seq":1866,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.053298+0000","last_change":"2024-07-26T23:35:17.404909+0000","last_active":"2024-07-26T23:58:48.053298+0000","last_peered":"2024-07-26T23:58:48.053298+0000","last_clean":"2024-07-26T23:58:48.053298+0000","last_became_active":"2024-07-26T23:35:16.707863+0000","last_became_peered":"2024-07-26T23:35:16.707863+0000","last_unstale":"2024-07-26T23:58:48.053298+0000","last_undegraded":"2024-07-26T23:58:48.053298+0000","last_fullsized":"2024-07-26T23:58:48.053298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_clean_scrub_stamp":"2024-07-26T23:35:01.337925+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:50:39.235610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":1869,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.122604+0000","last_change":"2024-07-26T23:35:22.241392+0000","last_active":"2024-07-26T23:58:45.122604+0000","last_peered":"2024-07-26T23:58:45.122604+0000","last_clean":"2024-07-26T23:58:45.122604+0000","last_became_active":"2024-07-26T23:35:19.087174+0000","last_became_peered":"2024-07-26T23:35:19.087174+0000","last_unstale":"2024-07-26T23:58:45.122604+0000","last_undegraded":"2024-07-26T23:58:45.122604+0000","last_fullsized":"2024-07-26T23:58:45.122604+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:54:22.975250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.074002362000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1855,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.060718+0000","last_change":"2024-07-26T23:36:39.448493+0000","last_active":"2024-07-26T23:58:48.060718+0000","last_peered":"2024-07-26T23:58:48.060718+0000","last_clean":"2024-07-26T23:58:48.060718+0000","last_became_active":"2024-07-26T23:36:39.441853+0000","last_became_peered":"2024-07-26T23:36:39.441853+0000","last_unstale":"2024-07-26T23:58:48.060718+0000","last_undegraded":"2024-07-26T23:58:48.060718+0000","last_fullsized":"2024-07-26T23:58:48.060718+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:33:54.532094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1868,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.094787+0000","last_change":"2024-07-26T23:35:22.212052+0000","last_active":"2024-07-26T23:58:45.094787+0000","last_peered":"2024-07-26T23:58:45.094787+0000","last_clean":"2024-07-26T23:58:45.094787+0000","last_became_active":"2024-07-26T23:35:17.846505+0000","last_became_peered":"2024-07-26T23:35:17.846505+0000","last_unstale":"2024-07-26T23:58:45.094787+0000","last_undegraded":"2024-07-26T23:58:45.094787+0000","last_fullsized":"2024-07-26T23:58:45.094787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:31.369623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11625252899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1840,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.185145+0000","last_change":"2024-07-26T23:36:33.996623+0000","last_active":"2024-07-26T23:58:45.185145+0000","last_peered":"2024-07-26T23:58:45.185145+0000","last_clean":"2024-07-26T23:58:45.185145+0000","last_became_active":"2024-07-26T23:36:33.958339+0000","last_became_peered":"2024-07-26T23:36:33.958339+0000","last_unstale":"2024-07-26T23:58:45.185145+0000","last_undegraded":"2024-07-26T23:58:45.185145+0000","last_fullsized":"2024-07-26T23:58:45.185145+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:19.336284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1880,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.013967+0000","last_change":"2024-07-26T23:35:22.093863+0000","last_active":"2024-07-26T23:58:48.013967+0000","last_peered":"2024-07-26T23:58:48.013967+0000","last_clean":"2024-07-26T23:58:48.013967+0000","last_became_active":"2024-07-26T23:35:17.814359+0000","last_became_peered":"2024-07-26T23:35:17.814359+0000","last_unstale":"2024-07-26T23:58:48.013967+0000","last_undegraded":"2024-07-26T23:58:48.013967+0000","last_fullsized":"2024-07-26T23:58:48.013967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:27:36.663020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060284956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.050550+0000","last_change":"2024-07-26T23:36:32.118711+0000","last_active":"2024-07-26T23:58:48.050550+0000","last_peered":"2024-07-26T23:58:48.050550+0000","last_clean":"2024-07-26T23:58:48.050550+0000","last_became_active":"2024-07-26T23:36:32.112540+0000","last_became_peered":"2024-07-26T23:36:32.112540+0000","last_unstale":"2024-07-26T23:58:48.050550+0000","last_undegraded":"2024-07-26T23:58:48.050550+0000","last_fullsized":"2024-07-26T23:58:48.050550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:32:07.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":1870,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.108137+0000","last_change":"2024-07-26T23:35:22.220704+0000","last_active":"2024-07-26T23:58:45.108137+0000","last_peered":"2024-07-26T23:58:45.108137+0000","last_clean":"2024-07-26T23:58:45.108137+0000","last_became_active":"2024-07-26T23:35:18.492082+0000","last_became_peered":"2024-07-26T23:35:18.492082+0000","last_unstale":"2024-07-26T23:58:45.108137+0000","last_undegraded":"2024-07-26T23:58:45.108137+0000","last_fullsized":"2024-07-26T23:58:45.108137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:00.865495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098681633000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1871,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.700871+0000","last_change":"2024-07-26T23:36:29.379730+0000","last_active":"2024-07-26T23:58:49.700871+0000","last_peered":"2024-07-26T23:58:49.700871+0000","last_clean":"2024-07-26T23:58:49.700871+0000","last_became_active":"2024-07-26T23:36:29.374355+0000","last_became_peered":"2024-07-26T23:36:29.374355+0000","last_unstale":"2024-07-26T23:58:49.700871+0000","last_undegraded":"2024-07-26T23:58:49.700871+0000","last_fullsized":"2024-07-26T23:58:49.700871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:55:50.796408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1875,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.135256+0000","last_change":"2024-07-26T23:35:22.376891+0000","last_active":"2024-07-26T23:58:46.135256+0000","last_peered":"2024-07-26T23:58:46.135256+0000","last_clean":"2024-07-26T23:58:46.135256+0000","last_became_active":"2024-07-26T23:35:18.507138+0000","last_became_peered":"2024-07-26T23:35:18.507138+0000","last_unstale":"2024-07-26T23:58:46.135256+0000","last_undegraded":"2024-07-26T23:58:46.135256+0000","last_fullsized":"2024-07-26T23:58:46.135256+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:56:22.382897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047472209000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.099229+0000","last_change":"2024-07-26T23:36:32.850160+0000","last_active":"2024-07-26T23:58:48.099229+0000","last_peered":"2024-07-26T23:58:48.099229+0000","last_clean":"2024-07-26T23:58:48.099229+0000","last_became_active":"2024-07-26T23:36:32.842079+0000","last_became_peered":"2024-07-26T23:36:32.842079+0000","last_unstale":"2024-07-26T23:58:48.099229+0000","last_undegraded":"2024-07-26T23:58:48.099229+0000","last_fullsized":"2024-07-26T23:58:48.099229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:53:03.317423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1867,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.151020+0000","last_change":"2024-07-26T23:35:22.251048+0000","last_active":"2024-07-26T23:58:45.151020+0000","last_peered":"2024-07-26T23:58:45.151020+0000","last_clean":"2024-07-26T23:58:45.151020+0000","last_became_active":"2024-07-26T23:35:17.925979+0000","last_became_peered":"2024-07-26T23:35:17.925979+0000","last_unstale":"2024-07-26T23:58:45.151020+0000","last_undegraded":"2024-07-26T23:58:45.151020+0000","last_fullsized":"2024-07-26T23:58:45.151020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:20.031998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073846415999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1857,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:47.993381+0000","last_change":"2024-07-26T23:36:31.631363+0000","last_active":"2024-07-26T23:58:47.993381+0000","last_peered":"2024-07-26T23:58:47.993381+0000","last_clean":"2024-07-26T23:58:47.993381+0000","last_became_active":"2024-07-26T23:36:31.543107+0000","last_became_peered":"2024-07-26T23:36:31.543107+0000","last_unstale":"2024-07-26T23:58:47.993381+0000","last_undegraded":"2024-07-26T23:58:47.993381+0000","last_fullsized":"2024-07-26T23:58:47.993381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:49:57.654672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1892,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.763722+0000","last_change":"2024-07-26T23:35:22.979978+0000","last_active":"2024-07-26T23:58:49.763722+0000","last_peered":"2024-07-26T23:58:49.763722+0000","last_clean":"2024-07-26T23:58:49.763722+0000","last_became_active":"2024-07-26T23:35:18.587303+0000","last_became_peered":"2024-07-26T23:35:18.587303+0000","last_unstale":"2024-07-26T23:58:49.763722+0000","last_undegraded":"2024-07-26T23:58:49.763722+0000","last_fullsized":"2024-07-26T23:58:49.763722+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:51:04.403633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.077704690000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1860,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.279717+0000","last_change":"2024-07-26T23:36:50.276152+0000","last_active":"2024-07-26T23:58:45.279717+0000","last_peered":"2024-07-26T23:58:45.279717+0000","last_clean":"2024-07-26T23:58:45.279717+0000","last_became_active":"2024-07-26T23:36:50.258118+0000","last_became_peered":"2024-07-26T23:36:50.258118+0000","last_unstale":"2024-07-26T23:58:45.279717+0000","last_undegraded":"2024-07-26T23:58:45.279717+0000","last_fullsized":"2024-07-26T23:58:45.279717+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:46:49.065664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1864,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.182051+0000","last_change":"2024-07-26T23:35:22.236663+0000","last_active":"2024-07-26T23:58:45.182051+0000","last_peered":"2024-07-26T23:58:45.182051+0000","last_clean":"2024-07-26T23:58:45.182051+0000","last_became_active":"2024-07-26T23:35:18.556903+0000","last_became_peered":"2024-07-26T23:35:18.556903+0000","last_unstale":"2024-07-26T23:58:45.182051+0000","last_undegraded":"2024-07-26T23:58:45.182051+0000","last_fullsized":"2024-07-26T23:58:45.182051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:27.775888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064553026999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1841,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.104035+0000","last_change":"2024-07-26T23:36:28.958153+0000","last_active":"2024-07-26T23:58:45.104035+0000","last_peered":"2024-07-26T23:58:45.104035+0000","last_clean":"2024-07-26T23:58:45.104035+0000","last_became_active":"2024-07-26T23:36:28.821361+0000","last_became_peered":"2024-07-26T23:36:28.821361+0000","last_unstale":"2024-07-26T23:58:45.104035+0000","last_undegraded":"2024-07-26T23:58:45.104035+0000","last_fullsized":"2024-07-26T23:58:45.104035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:05:13.869444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.427603+0000","last_change":"2024-07-26T23:36:29.506650+0000","last_active":"2024-07-26T23:58:49.427603+0000","last_peered":"2024-07-26T23:58:49.427603+0000","last_clean":"2024-07-26T23:58:49.427603+0000","last_became_active":"2024-07-26T23:36:29.496365+0000","last_became_peered":"2024-07-26T23:36:29.496365+0000","last_unstale":"2024-07-26T23:58:49.427603+0000","last_undegraded":"2024-07-26T23:58:49.427603+0000","last_fullsized":"2024-07-26T23:58:49.427603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:15:34.642884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.011318+0000","last_change":"2024-07-26T23:36:33.306681+0000","last_active":"2024-07-26T23:58:48.011318+0000","last_peered":"2024-07-26T23:58:48.011318+0000","last_clean":"2024-07-26T23:58:48.011318+0000","last_became_active":"2024-07-26T23:36:33.301227+0000","last_became_peered":"2024-07-26T23:36:33.301227+0000","last_unstale":"2024-07-26T23:58:48.011318+0000","last_undegraded":"2024-07-26T23:58:48.011318+0000","last_fullsized":"2024-07-26T23:58:48.011318+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:47:53.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:47.996084+0000","last_change":"2024-07-26T23:36:30.287713+0000","last_active":"2024-07-26T23:58:47.996084+0000","last_peered":"2024-07-26T23:58:47.996084+0000","last_clean":"2024-07-26T23:58:47.996084+0000","last_became_active":"2024-07-26T23:36:30.280064+0000","last_became_peered":"2024-07-26T23:36:30.280064+0000","last_unstale":"2024-07-26T23:58:47.996084+0000","last_undegraded":"2024-07-26T23:58:47.996084+0000","last_fullsized":"2024-07-26T23:58:47.996084+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:43:33.794147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1838,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.211236+0000","last_change":"2024-07-26T23:36:29.655104+0000","last_active":"2024-07-26T23:58:48.211236+0000","last_peered":"2024-07-26T23:58:48.211236+0000","last_clean":"2024-07-26T23:58:48.211236+0000","last_became_active":"2024-07-26T23:36:29.640586+0000","last_became_peered":"2024-07-26T23:36:29.640586+0000","last_unstale":"2024-07-26T23:58:48.211236+0000","last_undegraded":"2024-07-26T23:58:48.211236+0000","last_fullsized":"2024-07-26T23:58:48.211236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:04:00.672728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.441671+0000","last_change":"2024-07-26T23:36:28.901469+0000","last_active":"2024-07-26T23:58:49.441671+0000","last_peered":"2024-07-26T23:58:49.441671+0000","last_clean":"2024-07-26T23:58:49.441671+0000","last_became_active":"2024-07-26T23:36:28.852843+0000","last_became_peered":"2024-07-26T23:36:28.852843+0000","last_unstale":"2024-07-26T23:58:49.441671+0000","last_undegraded":"2024-07-26T23:58:49.441671+0000","last_fullsized":"2024-07-26T23:58:49.441671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:09.082990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1839,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.176565+0000","last_change":"2024-07-26T23:36:31.130573+0000","last_active":"2024-07-26T23:58:48.176565+0000","last_peered":"2024-07-26T23:58:48.176565+0000","last_clean":"2024-07-26T23:58:48.176565+0000","last_became_active":"2024-07-26T23:36:31.124146+0000","last_became_peered":"2024-07-26T23:36:31.124146+0000","last_unstale":"2024-07-26T23:58:48.176565+0000","last_undegraded":"2024-07-26T23:58:48.176565+0000","last_fullsized":"2024-07-26T23:58:48.176565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:07:28.551245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1842,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.090103+0000","last_change":"2024-07-26T23:36:30.187288+0000","last_active":"2024-07-26T23:58:45.090103+0000","last_peered":"2024-07-26T23:58:45.090103+0000","last_clean":"2024-07-26T23:58:45.090103+0000","last_became_active":"2024-07-26T23:36:30.173625+0000","last_became_peered":"2024-07-26T23:36:30.173625+0000","last_unstale":"2024-07-26T23:58:45.090103+0000","last_undegraded":"2024-07-26T23:58:45.090103+0000","last_fullsized":"2024-07-26T23:58:45.090103+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:55:38.891400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1851,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.119799+0000","last_change":"2024-07-26T23:36:31.996757+0000","last_active":"2024-07-26T23:58:46.119799+0000","last_peered":"2024-07-26T23:58:46.119799+0000","last_clean":"2024-07-26T23:58:46.119799+0000","last_became_active":"2024-07-26T23:36:31.990984+0000","last_became_peered":"2024-07-26T23:36:31.990984+0000","last_unstale":"2024-07-26T23:58:46.119799+0000","last_undegraded":"2024-07-26T23:58:46.119799+0000","last_fullsized":"2024-07-26T23:58:46.119799+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:24:44.872192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1870,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.698237+0000","last_change":"2024-07-26T23:36:33.389570+0000","last_active":"2024-07-26T23:58:49.698237+0000","last_peered":"2024-07-26T23:58:49.698237+0000","last_clean":"2024-07-26T23:58:49.698237+0000","last_became_active":"2024-07-26T23:36:33.382005+0000","last_became_peered":"2024-07-26T23:36:33.382005+0000","last_unstale":"2024-07-26T23:58:49.698237+0000","last_undegraded":"2024-07-26T23:58:49.698237+0000","last_fullsized":"2024-07-26T23:58:49.698237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:33:51.810118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.430486+0000","last_change":"2024-07-26T23:36:28.719222+0000","last_active":"2024-07-26T23:58:49.430486+0000","last_peered":"2024-07-26T23:58:49.430486+0000","last_clean":"2024-07-26T23:58:49.430486+0000","last_became_active":"2024-07-26T23:36:28.654711+0000","last_became_peered":"2024-07-26T23:36:28.654711+0000","last_unstale":"2024-07-26T23:58:49.430486+0000","last_undegraded":"2024-07-26T23:58:49.430486+0000","last_fullsized":"2024-07-26T23:58:49.430486+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:06.380712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1859,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-07-26T23:58:45.267443+0000","last_change":"2024-07-26T23:36:36.812820+0000","last_active":"2024-07-26T23:58:45.267443+0000","last_peered":"2024-07-26T23:58:45.267443+0000","last_clean":"2024-07-26T23:58:45.267443+0000","last_became_active":"2024-07-26T23:36:36.803418+0000","last_became_peered":"2024-07-26T23:36:36.803418+0000","last_unstale":"2024-07-26T23:58:45.267443+0000","last_undegraded":"2024-07-26T23:58:45.267443+0000","last_fullsized":"2024-07-26T23:58:45.267443+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:26:11.430116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":21,"seq":90194313503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94716,"kb_used_data":14952,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94277124,"statfs":{"total":96636764160,"available":96539774976,"internally_reserved":0,"allocated":15310848,"data_stored":9567401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313506,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90608,"kb_used_data":14992,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281232,"statfs":{"total":96636764160,"available":96543981568,"internally_reserved":0,"allocated":15351808,"data_stored":9629741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[]},{"osd":3,"up_from":20,"seq":85899346207,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90584,"kb_used_data":14956,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281256,"statfs":{"total":96636764160,"available":96544006144,"internally_reserved":0,"allocated":15314944,"data_stored":9588800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346211,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91020,"kb_used_data":15404,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94280820,"statfs":{"total":96636764160,"available":96543559680,"internally_reserved":0,"allocated":15773696,"data_stored":10031266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411616,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82248,"kb_used_data":14964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94289592,"statfs":{"total":96636764160,"available":96552542208,"internally_reserved":0,"allocated":15323136,"data_stored":9574126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509728,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90672,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281168,"statfs":{"total":96636764160,"available":96543916032,"internally_reserved":0,"allocated":15429632,"data_stored":9694861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":110,"apply_latency_ms":110,"commit_latency_ns":110000000,"apply_latency_ns":110000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575134,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78052,"kb_used_data":14940,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94293788,"statfs":{"total":96636764160,"available":96556838912,"internally_reserved":0,"allocated":15298560,"data_stored":9568325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575133,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95140,"kb_used_data":15352,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94276700,"statfs":{"total":96636764160,"available":96539340800,"internally_reserved":0,"allocated":15720448,"data_stored":9990675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T23:58:52.185 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T23:58:52.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.271+0000 7f8116a11640 10 monclient: get_monmap_and_config 2024-07-26T23:58:52.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.271+0000 7f8116a11640 10 monclient: build_initial_monmap 2024-07-26T23:58:52.286 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.271+0000 7f8116a11640 10 monclient: monmap: 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:52.272540+0000 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:52.272540+0000 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:52.287 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:52.288 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:52.288 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 1 Processor -- start 2024-07-26T23:58:52.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 1 -- start start 2024-07-26T23:58:52.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient: init 2024-07-26T23:58:52.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:52.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-07-26T23:58:52.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f81100b4bb0 0x7f811006e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient(hunting): picked mon.noname-b con 0x7f81100b4bb0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:52.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f811006f2a0 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:52.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8110141340 0x7f811014e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient(hunting): picked mon.noname-a con 0x7f8110141340 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:52.293 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81100b0960 con 0x7f811006f2a0 2024-07-26T23:58:52.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f81100b4f90 con 0x7f8110141340 2024-07-26T23:58:52.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f8110074410 con 0x7f81100b4bb0 2024-07-26T23:58:52.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:52.294 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.272+0000 7f8116a11640 10 monclient(hunting): authenticate will time out at 2765.308939s 2024-07-26T23:58:52.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f810f7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f811006f2a0 0x7f8110140c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:53888/0 (socket says 172.21.15.139:53888) 2024-07-26T23:58:52.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f810f7fe640 1 -- 172.21.15.139:0/706241344 learned_addr learned my addr 172.21.15.139:0/706241344 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:58:52.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f810ffff640 1 --2- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f81100b4bb0 0x7f811006e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f8114fae640 1 --2- 172.21.15.139:0/706241344 >> v2:172.21.15.139:3300/0 conn(0x7f8110141340 0x7f811014e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.295 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f8114fae640 10 monclient(hunting): get_auth_request con 0x7f8110141340 auth_method 0 2024-07-26T23:58:52.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f8114fae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f8114fae640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f8114fae640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.296 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f810ffff640 10 monclient(hunting): get_auth_request con 0x7f81100b4bb0 auth_method 0 2024-07-26T23:58:52.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f810ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f810ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.273+0000 7f810ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.274+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.297 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.274+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.274+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.274+0000 7f8114fae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.274+0000 7f8114fae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.298 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.274+0000 7f8114fae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.275+0000 7f810effd640 1 -- 172.21.15.139:0/706241344 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400644252 0 0) 0x7f81100b0960 con 0x7f811006f2a0 2024-07-26T23:58:52.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.275+0000 7f810effd640 10 monclient(hunting): _negotiate 2024-07-26T23:58:52.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.275+0000 7f810effd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.275+0000 7f810effd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.299 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.275+0000 7f810effd640 10 monclient(hunting): my global_id is 20254 2024-07-26T23:58:52.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.275+0000 7f810effd640 1 -- 172.21.15.139:0/706241344 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80f8003520 con 0x7f811006f2a0 2024-07-26T23:58:52.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 10 monclient(hunting): handle_auth_done global_id 15522 payload 274 2024-07-26T23:58:52.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 1 -- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f81100b4bb0 msgr2=0x7f811006e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:52.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 1 --2- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f81100b4bb0 0x7f811006e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 1 -- 172.21.15.139:0/706241344 >> v1:172.21.15.177:6789/0 conn(0x7f811006f2a0 legacy=0x7f8110140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 10 monclient: _finish_hunting 0 2024-07-26T23:58:52.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 1 monclient: found mon.noname-a 2024-07-26T23:58:52.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:52.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:52.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f8114fae640 1 -- 172.21.15.139:0/706241344 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8110074610 con 0x7f8110141340 2024-07-26T23:58:52.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.276+0000 7f810ffff640 1 --2- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f81100b4bb0 0x7f811006e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T23:58:52.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.278+0000 7f8114fae640 1 --2- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8110141340 0x7f811014e290 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f8108009d90 tx=0x7f810802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5213ca285a2c83ff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:52.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 1 -- 172.21.15.139:0/706241344 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108039070 con 0x7f8110141340 2024-07-26T23:58:52.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:52.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: dump: 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: _finish_auth 0 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: _check_auth_tickets 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:52.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 1 -- 172.21.15.139:0/706241344 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f810802cbc0 con 0x7f8110141340 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f8116a11640 5 monclient: authenticate success, global_id 15522 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f8116a11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f8116a11640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f8116a11640 4 set_mon_vals no callback set 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 1 -- 172.21.15.139:0/706241344 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108034440 con 0x7f8110141340 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.281+0000 7f810effd640 10 monclient: dump: 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.308 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 10 monclient: shutdown 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 -- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8110141340 msgr2=0x7f811014e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 --2- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8110141340 0x7f811014e290 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f8108009d90 tx=0x7f810802be10 comp rx=0 tx=0).stop 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 -- 172.21.15.139:0/706241344 shutdown_connections 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 --2- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8110141340 0x7f811014e290 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 --2- 172.21.15.139:0/706241344 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f81100b4bb0 0x7f811006e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 -- 172.21.15.139:0/706241344 >> 172.21.15.139:0/706241344 conn(0x7f81100b7d60 msgr2=0x7f81100b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 -- 172.21.15.139:0/706241344 shutdown_connections 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.282+0000 7f8116a11640 1 -- 172.21.15.139:0/706241344 wait complete. 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.283+0000 7f8116a11640 10 monclient: build_initial_monmap 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.283+0000 7f8116a11640 10 monclient: monmap: 2024-07-26T23:58:52.310 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.283+0000 7f8116a11640 1 librados: starting msgr at 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.283+0000 7f8116a11640 1 librados: starting objecter 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 Processor -- start 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 -- start start 2024-07-26T23:58:52.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 librados: setting wanted keys 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 librados: calling monclient init 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient: init 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81100b4bb0 0x7f8110144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): picked mon.noname-a con 0x7f81100b4bb0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8110141340 0x7f8110144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): picked mon.noname-c con 0x7f8110141340 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8110147510 0x7f8110147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): picked mon.noname-b con 0x7f8110147510 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:52.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f811006ed80 con 0x7f81100b4bb0 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81100b0d80 con 0x7f8110147510 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f8110147ea0 con 0x7f8110141340 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.284+0000 7f8116a11640 10 monclient(hunting): authenticate will time out at 2765.320814s 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810ffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81100b4bb0 0x7f8110144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810f7fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8110141340 0x7f8110144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f8114fae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8110147510 0x7f8110147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f8114fae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8110147510 0x7f8110147960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:46816/0 (socket says 172.21.15.139:46816) 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810ffff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81100b4bb0 0x7f8110144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36250/0 (socket says 172.21.15.139:36250) 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f8114fae640 1 -- 172.21.15.139:0/3655229315 learned_addr learned my addr 172.21.15.139:0/3655229315 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:52.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f8114fae640 10 monclient(hunting): get_auth_request con 0x7f8110147510 auth_method 0 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f8114fae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f8114fae640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f8114fae640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810ffff640 10 monclient(hunting): get_auth_request con 0x7f81100b4bb0 auth_method 0 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810f7fe640 10 monclient(hunting): get_auth_request con 0x7f8110141340 auth_method 0 2024-07-26T23:58:52.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.285+0000 7f810f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f8114fae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f8114fae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f8114fae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f810ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.286+0000 7f810f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.288+0000 7f810ffff640 10 monclient(hunting): handle_auth_done global_id 15525 payload 824 2024-07-26T23:58:52.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.288+0000 7f810ffff640 1 -- 172.21.15.139:0/3655229315 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8110141340 msgr2=0x7f8110144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.288+0000 7f810ffff640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8110141340 0x7f8110144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.288+0000 7f810ffff640 1 -- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8110147510 msgr2=0x7f8110147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.288+0000 7f810ffff640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8110147510 0x7f8110147960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.317 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.288+0000 7f810ffff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:52.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.288+0000 7f810ffff640 1 monclient: found mon.noname-a 2024-07-26T23:58:52.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.289+0000 7f810ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:52.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.289+0000 7f810ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:52.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.289+0000 7f810ffff640 1 -- 172.21.15.139:0/3655229315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81101466e0 con 0x7f81100b4bb0 2024-07-26T23:58:52.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.291+0000 7f810ffff640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81100b4bb0 0x7f8110144f70 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f810000e900 tx=0x7f810000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:52.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.291+0000 7f810cff9640 10 client.?.objecter ms_handle_connect 0x7f81100b4bb0 2024-07-26T23:58:52.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.291+0000 7f810cff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:52.321 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 1 -- 172.21.15.139:0/3655229315 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100017070 con 0x7f81100b4bb0 2024-07-26T23:58:52.321 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:52.322 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: dump: 2024-07-26T23:58:52.322 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.322 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.323 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.323 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.323 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.323 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.323 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.324 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.324 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.324 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: _finish_auth 0 2024-07-26T23:58:52.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: _check_auth_tickets 2024-07-26T23:58:52.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:52.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 1 -- 172.21.15.139:0/3655229315 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f810000ca00 con 0x7f81100b4bb0 2024-07-26T23:58:52.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 5 monclient: authenticate success, global_id 15525 2024-07-26T23:58:52.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 10 monclient: _renew_subs 2024-07-26T23:58:52.325 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:52.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8110146950 con 0x7f81100b4bb0 2024-07-26T23:58:52.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:52.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 1 -- 172.21.15.139:0/3655229315 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810000cdf0 con 0x7f81100b4bb0 2024-07-26T23:58:52.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:52.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810cff9640 10 monclient: dump: 2024-07-26T23:58:52.327 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.327 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.327 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.327 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.327 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.328 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.328 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.328 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.328 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.328 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 10 client.15525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:52.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 10 monclient: _renew_subs 2024-07-26T23:58:52.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:52.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f810e7fc640 4 set_mon_vals no callback set 2024-07-26T23:58:52.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81101d8f20 con 0x7f81100b4bb0 2024-07-26T23:58:52.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.294+0000 7f8116a11640 1 librados: init done 2024-07-26T23:58:52.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.295+0000 7f8116a11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:52.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.295+0000 7f8116a11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:52.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.295+0000 7f8116a11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:52.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.295+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f811006e630 con 0x7f81100b4bb0 2024-07-26T23:58:52.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.316+0000 7f810cff9640 1 -- 172.21.15.139:0/3655229315 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f810001a070 con 0x7f81100b4bb0 2024-07-26T23:58:52.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.317+0000 7f810cff9640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f80fc0789a0 0x7f80fc07ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.317+0000 7f810f7fe640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f80fc0789a0 0x7f80fc07ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.317+0000 7f810f7fe640 10 monclient: get_auth_request con 0x7f80fc0789a0 auth_method 0 2024-07-26T23:58:52.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.317+0000 7f810cff9640 10 client.15525.objecter ms_handle_connect 0x7f80fc0789a0 2024-07-26T23:58:52.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.317+0000 7f810f7fe640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f80fc0789a0 0x7f80fc07ae60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8104007f50 tx=0x7f8104007ee0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:52.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.319+0000 7f810cff9640 1 -- 172.21.15.139:0/3655229315 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(969..969 src has 501..969) ==== 6365+0+0 (secure 0 0 0) 0x7f8100014db0 con 0x7f81100b4bb0 2024-07-26T23:58:52.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.319+0000 7f810cff9640 10 client.15525.objecter ms_dispatch 0x7f8110000b90 osd_map(969..969 src has 501..969) 2024-07-26T23:58:52.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.319+0000 7f810cff9640 3 client.15525.objecter handle_osd_map got epochs [969,969] > 0 2024-07-26T23:58:52.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.319+0000 7f810cff9640 3 client.15525.objecter handle_osd_map decoding full epoch 969 2024-07-26T23:58:52.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.319+0000 7f810cff9640 20 client.15525.objecter dump_active .. 0 homeless 2024-07-26T23:58:52.570 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.555+0000 7f810cff9640 1 -- 172.21.15.139:0/3655229315 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f81000619a0 con 0x7f81100b4bb0 2024-07-26T23:58:52.571 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.555+0000 7f810cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:52.571 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.555+0000 7f810cff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:52.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.739+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f81101d8cd0 con 0x7f80fc0789a0 2024-07-26T23:58:52.756 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.741+0000 7f810cff9640 1 -- 172.21.15.139:0/3655229315 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116459 (secure 0 0 0) 0x7f8104001430 con 0x7f80fc0789a0 2024-07-26T23:58:52.757 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:58:52.758 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 10 librados: watch_flush enter 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 10 librados: watch_flush exit 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 20 client.15525.objecter shutdown clearing up homeless session... 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 10 client.15525.objecter successfully canceled tick 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f80fc0789a0 msgr2=0x7f80fc07ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f80fc0789a0 0x7f80fc07ae60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8104007f50 tx=0x7f8104007ee0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 10 monclient: shutdown 2024-07-26T23:58:52.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81100b4bb0 msgr2=0x7f8110144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81100b4bb0 0x7f8110144f70 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f810000e900 tx=0x7f810000edc0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 shutdown_connections 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f80fc0789a0 0x7f80fc07ae60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8110141340 0x7f8110144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8110147510 0x7f8110147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 --2- 172.21.15.139:0/3655229315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f81100b4bb0 0x7f8110144f70 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 >> 172.21.15.139:0/3655229315 conn(0x7f81100b7d60 msgr2=0x7f81100b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 shutdown_connections 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 -- 172.21.15.139:0/3655229315 wait complete. 2024-07-26T23:58:52.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.744+0000 7f8116a11640 1 librados: shutdown 2024-07-26T23:58:52.771 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":1688,"stamp":"2024-07-26T23:58:52.216184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":52,"num_read_kb":43,"num_write":66,"num_write_kb":593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":713056,"kb_used_data":120644,"kb_used_omap":12,"kb_used_meta":592179,"kb_avail":754261664,"statfs":{"total":773094113280,"available":772363943936,"internally_reserved":0,"allocated":123539456,"data_stored":77580750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":606391856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":432,"apply_latency_ms":432,"commit_latency_ns":432000000,"apply_latency_ns":432000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.004226"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1850,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.132348+0000","last_change":"2024-07-26T23:36:37.015743+0000","last_active":"2024-07-26T23:58:46.132348+0000","last_peered":"2024-07-26T23:58:46.132348+0000","last_clean":"2024-07-26T23:58:46.132348+0000","last_became_active":"2024-07-26T23:36:37.010177+0000","last_became_peered":"2024-07-26T23:36:37.010177+0000","last_unstale":"2024-07-26T23:58:46.132348+0000","last_undegraded":"2024-07-26T23:58:46.132348+0000","last_fullsized":"2024-07-26T23:58:46.132348+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:54:54.240713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1868,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.841031+0000","last_change":"2024-07-26T23:36:32.386250+0000","last_active":"2024-07-26T23:58:50.841031+0000","last_peered":"2024-07-26T23:58:50.841031+0000","last_clean":"2024-07-26T23:58:50.841031+0000","last_became_active":"2024-07-26T23:36:32.380680+0000","last_became_peered":"2024-07-26T23:36:32.380680+0000","last_unstale":"2024-07-26T23:58:50.841031+0000","last_undegraded":"2024-07-26T23:58:50.841031+0000","last_fullsized":"2024-07-26T23:58:50.841031+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:19:53.312577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1846,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.439029+0000","last_change":"2024-07-26T23:36:32.478927+0000","last_active":"2024-07-26T23:58:49.439029+0000","last_peered":"2024-07-26T23:58:49.439029+0000","last_clean":"2024-07-26T23:58:49.439029+0000","last_became_active":"2024-07-26T23:36:32.473374+0000","last_became_peered":"2024-07-26T23:36:32.473374+0000","last_unstale":"2024-07-26T23:58:49.439029+0000","last_undegraded":"2024-07-26T23:58:49.439029+0000","last_fullsized":"2024-07-26T23:58:49.439029+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:13:31.972041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.701385+0000","last_change":"2024-07-26T23:36:45.735883+0000","last_active":"2024-07-26T23:58:49.701385+0000","last_peered":"2024-07-26T23:58:49.701385+0000","last_clean":"2024-07-26T23:58:49.701385+0000","last_became_active":"2024-07-26T23:36:45.718574+0000","last_became_peered":"2024-07-26T23:36:45.718574+0000","last_unstale":"2024-07-26T23:58:49.701385+0000","last_undegraded":"2024-07-26T23:58:49.701385+0000","last_fullsized":"2024-07-26T23:58:49.701385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:22:53.643566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1851,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.711009+0000","last_change":"2024-07-26T23:36:33.937849+0000","last_active":"2024-07-26T23:58:49.711009+0000","last_peered":"2024-07-26T23:58:49.711009+0000","last_clean":"2024-07-26T23:58:49.711009+0000","last_became_active":"2024-07-26T23:36:33.915707+0000","last_became_peered":"2024-07-26T23:36:33.915707+0000","last_unstale":"2024-07-26T23:58:49.711009+0000","last_undegraded":"2024-07-26T23:58:49.711009+0000","last_fullsized":"2024-07-26T23:58:49.711009+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.705473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"44'7","reported_seq":1854,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.005889+0000","last_change":"2024-07-26T23:36:30.524945+0000","last_active":"2024-07-26T23:58:48.005889+0000","last_peered":"2024-07-26T23:58:48.005889+0000","last_clean":"2024-07-26T23:58:48.005889+0000","last_became_active":"2024-07-26T23:36:30.507651+0000","last_became_peered":"2024-07-26T23:36:30.507651+0000","last_unstale":"2024-07-26T23:58:48.005889+0000","last_undegraded":"2024-07-26T23:58:48.005889+0000","last_fullsized":"2024-07-26T23:58:48.005889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:36:56.132284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1845,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.428542+0000","last_change":"2024-07-26T23:36:29.482671+0000","last_active":"2024-07-26T23:58:49.428542+0000","last_peered":"2024-07-26T23:58:49.428542+0000","last_clean":"2024-07-26T23:58:49.428542+0000","last_became_active":"2024-07-26T23:36:29.468516+0000","last_became_peered":"2024-07-26T23:36:29.468516+0000","last_unstale":"2024-07-26T23:58:49.428542+0000","last_undegraded":"2024-07-26T23:58:49.428542+0000","last_fullsized":"2024-07-26T23:58:49.428542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:10:56.313880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1871,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.705157+0000","last_change":"2024-07-26T23:36:50.102068+0000","last_active":"2024-07-26T23:58:49.705157+0000","last_peered":"2024-07-26T23:58:49.705157+0000","last_clean":"2024-07-26T23:58:49.705157+0000","last_became_active":"2024-07-26T23:36:50.085394+0000","last_became_peered":"2024-07-26T23:36:50.085394+0000","last_unstale":"2024-07-26T23:58:49.705157+0000","last_undegraded":"2024-07-26T23:58:49.705157+0000","last_fullsized":"2024-07-26T23:58:49.705157+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:03:21.264586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1845,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.436927+0000","last_change":"2024-07-26T23:36:32.322928+0000","last_active":"2024-07-26T23:58:49.436927+0000","last_peered":"2024-07-26T23:58:49.436927+0000","last_clean":"2024-07-26T23:58:49.436927+0000","last_became_active":"2024-07-26T23:36:32.315924+0000","last_became_peered":"2024-07-26T23:36:32.315924+0000","last_unstale":"2024-07-26T23:58:49.436927+0000","last_undegraded":"2024-07-26T23:58:49.436927+0000","last_fullsized":"2024-07-26T23:58:49.436927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:29:47.697657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.737182+0000","last_change":"2024-07-26T23:36:33.078206+0000","last_active":"2024-07-26T23:58:49.737182+0000","last_peered":"2024-07-26T23:58:49.737182+0000","last_clean":"2024-07-26T23:58:49.737182+0000","last_became_active":"2024-07-26T23:36:33.069066+0000","last_became_peered":"2024-07-26T23:36:33.069066+0000","last_unstale":"2024-07-26T23:58:49.737182+0000","last_undegraded":"2024-07-26T23:58:49.737182+0000","last_fullsized":"2024-07-26T23:58:49.737182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:58:53.548672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1870,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.846114+0000","last_change":"2024-07-26T23:36:32.994273+0000","last_active":"2024-07-26T23:58:50.846114+0000","last_peered":"2024-07-26T23:58:50.846114+0000","last_clean":"2024-07-26T23:58:50.846114+0000","last_became_active":"2024-07-26T23:36:32.984857+0000","last_became_peered":"2024-07-26T23:36:32.984857+0000","last_unstale":"2024-07-26T23:58:50.846114+0000","last_undegraded":"2024-07-26T23:58:50.846114+0000","last_fullsized":"2024-07-26T23:58:50.846114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:54:04.890736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.141761+0000","last_change":"2024-07-26T23:36:30.566094+0000","last_active":"2024-07-26T23:58:48.141761+0000","last_peered":"2024-07-26T23:58:48.141761+0000","last_clean":"2024-07-26T23:58:48.141761+0000","last_became_active":"2024-07-26T23:36:30.560131+0000","last_became_peered":"2024-07-26T23:36:30.560131+0000","last_unstale":"2024-07-26T23:58:48.141761+0000","last_undegraded":"2024-07-26T23:58:48.141761+0000","last_fullsized":"2024-07-26T23:58:48.141761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:58:36.316935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.695745+0000","last_change":"2024-07-26T23:36:44.212854+0000","last_active":"2024-07-26T23:58:49.695745+0000","last_peered":"2024-07-26T23:58:49.695745+0000","last_clean":"2024-07-26T23:58:49.695745+0000","last_became_active":"2024-07-26T23:36:44.149525+0000","last_became_peered":"2024-07-26T23:36:44.149525+0000","last_unstale":"2024-07-26T23:58:49.695745+0000","last_undegraded":"2024-07-26T23:58:49.695745+0000","last_fullsized":"2024-07-26T23:58:49.695745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:09:35.012505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'32","reported_seq":1866,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.053298+0000","last_change":"2024-07-26T23:35:17.404909+0000","last_active":"2024-07-26T23:58:48.053298+0000","last_peered":"2024-07-26T23:58:48.053298+0000","last_clean":"2024-07-26T23:58:48.053298+0000","last_became_active":"2024-07-26T23:35:16.707863+0000","last_became_peered":"2024-07-26T23:35:16.707863+0000","last_unstale":"2024-07-26T23:58:48.053298+0000","last_undegraded":"2024-07-26T23:58:48.053298+0000","last_fullsized":"2024-07-26T23:58:48.053298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_clean_scrub_stamp":"2024-07-26T23:35:01.337925+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:50:39.235610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":1877,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.688571+0000","last_change":"2024-07-26T23:35:22.241392+0000","last_active":"2024-07-26T23:58:49.688571+0000","last_peered":"2024-07-26T23:58:49.688571+0000","last_clean":"2024-07-26T23:58:49.688571+0000","last_became_active":"2024-07-26T23:35:19.087174+0000","last_became_peered":"2024-07-26T23:35:19.087174+0000","last_unstale":"2024-07-26T23:58:49.688571+0000","last_undegraded":"2024-07-26T23:58:49.688571+0000","last_fullsized":"2024-07-26T23:58:49.688571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:54:22.975250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.074002362000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1855,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.060718+0000","last_change":"2024-07-26T23:36:39.448493+0000","last_active":"2024-07-26T23:58:48.060718+0000","last_peered":"2024-07-26T23:58:48.060718+0000","last_clean":"2024-07-26T23:58:48.060718+0000","last_became_active":"2024-07-26T23:36:39.441853+0000","last_became_peered":"2024-07-26T23:36:39.441853+0000","last_unstale":"2024-07-26T23:58:48.060718+0000","last_undegraded":"2024-07-26T23:58:48.060718+0000","last_fullsized":"2024-07-26T23:58:48.060718+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:33:54.532094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.682140+0000","last_change":"2024-07-26T23:35:22.212052+0000","last_active":"2024-07-26T23:58:49.682140+0000","last_peered":"2024-07-26T23:58:49.682140+0000","last_clean":"2024-07-26T23:58:49.682140+0000","last_became_active":"2024-07-26T23:35:17.846505+0000","last_became_peered":"2024-07-26T23:35:17.846505+0000","last_unstale":"2024-07-26T23:58:49.682140+0000","last_undegraded":"2024-07-26T23:58:49.682140+0000","last_fullsized":"2024-07-26T23:58:49.682140+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:31.369623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11625252899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1848,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.705039+0000","last_change":"2024-07-26T23:36:33.996623+0000","last_active":"2024-07-26T23:58:49.705039+0000","last_peered":"2024-07-26T23:58:49.705039+0000","last_clean":"2024-07-26T23:58:49.705039+0000","last_became_active":"2024-07-26T23:36:33.958339+0000","last_became_peered":"2024-07-26T23:36:33.958339+0000","last_unstale":"2024-07-26T23:58:49.705039+0000","last_undegraded":"2024-07-26T23:58:49.705039+0000","last_fullsized":"2024-07-26T23:58:49.705039+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:19.336284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1880,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.013967+0000","last_change":"2024-07-26T23:35:22.093863+0000","last_active":"2024-07-26T23:58:48.013967+0000","last_peered":"2024-07-26T23:58:48.013967+0000","last_clean":"2024-07-26T23:58:48.013967+0000","last_became_active":"2024-07-26T23:35:17.814359+0000","last_became_peered":"2024-07-26T23:35:17.814359+0000","last_unstale":"2024-07-26T23:58:48.013967+0000","last_undegraded":"2024-07-26T23:58:48.013967+0000","last_fullsized":"2024-07-26T23:58:48.013967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:27:36.663020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060284956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.050550+0000","last_change":"2024-07-26T23:36:32.118711+0000","last_active":"2024-07-26T23:58:48.050550+0000","last_peered":"2024-07-26T23:58:48.050550+0000","last_clean":"2024-07-26T23:58:48.050550+0000","last_became_active":"2024-07-26T23:36:32.112540+0000","last_became_peered":"2024-07-26T23:36:32.112540+0000","last_unstale":"2024-07-26T23:58:48.050550+0000","last_undegraded":"2024-07-26T23:58:48.050550+0000","last_fullsized":"2024-07-26T23:58:48.050550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:32:07.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":1878,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.692788+0000","last_change":"2024-07-26T23:35:22.220704+0000","last_active":"2024-07-26T23:58:49.692788+0000","last_peered":"2024-07-26T23:58:49.692788+0000","last_clean":"2024-07-26T23:58:49.692788+0000","last_became_active":"2024-07-26T23:35:18.492082+0000","last_became_peered":"2024-07-26T23:35:18.492082+0000","last_unstale":"2024-07-26T23:58:49.692788+0000","last_undegraded":"2024-07-26T23:58:49.692788+0000","last_fullsized":"2024-07-26T23:58:49.692788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:00.865495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098681633000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1871,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.700871+0000","last_change":"2024-07-26T23:36:29.379730+0000","last_active":"2024-07-26T23:58:49.700871+0000","last_peered":"2024-07-26T23:58:49.700871+0000","last_clean":"2024-07-26T23:58:49.700871+0000","last_became_active":"2024-07-26T23:36:29.374355+0000","last_became_peered":"2024-07-26T23:36:29.374355+0000","last_unstale":"2024-07-26T23:58:49.700871+0000","last_undegraded":"2024-07-26T23:58:49.700871+0000","last_fullsized":"2024-07-26T23:58:49.700871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:55:50.796408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1875,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.135256+0000","last_change":"2024-07-26T23:35:22.376891+0000","last_active":"2024-07-26T23:58:46.135256+0000","last_peered":"2024-07-26T23:58:46.135256+0000","last_clean":"2024-07-26T23:58:46.135256+0000","last_became_active":"2024-07-26T23:35:18.507138+0000","last_became_peered":"2024-07-26T23:35:18.507138+0000","last_unstale":"2024-07-26T23:58:46.135256+0000","last_undegraded":"2024-07-26T23:58:46.135256+0000","last_fullsized":"2024-07-26T23:58:46.135256+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:56:22.382897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047472209000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.099229+0000","last_change":"2024-07-26T23:36:32.850160+0000","last_active":"2024-07-26T23:58:48.099229+0000","last_peered":"2024-07-26T23:58:48.099229+0000","last_clean":"2024-07-26T23:58:48.099229+0000","last_became_active":"2024-07-26T23:36:32.842079+0000","last_became_peered":"2024-07-26T23:36:32.842079+0000","last_unstale":"2024-07-26T23:58:48.099229+0000","last_undegraded":"2024-07-26T23:58:48.099229+0000","last_fullsized":"2024-07-26T23:58:48.099229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:53:03.317423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1875,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.697649+0000","last_change":"2024-07-26T23:35:22.251048+0000","last_active":"2024-07-26T23:58:49.697649+0000","last_peered":"2024-07-26T23:58:49.697649+0000","last_clean":"2024-07-26T23:58:49.697649+0000","last_became_active":"2024-07-26T23:35:17.925979+0000","last_became_peered":"2024-07-26T23:35:17.925979+0000","last_unstale":"2024-07-26T23:58:49.697649+0000","last_undegraded":"2024-07-26T23:58:49.697649+0000","last_fullsized":"2024-07-26T23:58:49.697649+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:20.031998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073846415999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1857,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:47.993381+0000","last_change":"2024-07-26T23:36:31.631363+0000","last_active":"2024-07-26T23:58:47.993381+0000","last_peered":"2024-07-26T23:58:47.993381+0000","last_clean":"2024-07-26T23:58:47.993381+0000","last_became_active":"2024-07-26T23:36:31.543107+0000","last_became_peered":"2024-07-26T23:36:31.543107+0000","last_unstale":"2024-07-26T23:58:47.993381+0000","last_undegraded":"2024-07-26T23:58:47.993381+0000","last_fullsized":"2024-07-26T23:58:47.993381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:49:57.654672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1892,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.763722+0000","last_change":"2024-07-26T23:35:22.979978+0000","last_active":"2024-07-26T23:58:49.763722+0000","last_peered":"2024-07-26T23:58:49.763722+0000","last_clean":"2024-07-26T23:58:49.763722+0000","last_became_active":"2024-07-26T23:35:18.587303+0000","last_became_peered":"2024-07-26T23:35:18.587303+0000","last_unstale":"2024-07-26T23:58:49.763722+0000","last_undegraded":"2024-07-26T23:58:49.763722+0000","last_fullsized":"2024-07-26T23:58:49.763722+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:51:04.403633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.077704690000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1870,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.825199+0000","last_change":"2024-07-26T23:36:50.276152+0000","last_active":"2024-07-26T23:58:50.825199+0000","last_peered":"2024-07-26T23:58:50.825199+0000","last_clean":"2024-07-26T23:58:50.825199+0000","last_became_active":"2024-07-26T23:36:50.258118+0000","last_became_peered":"2024-07-26T23:36:50.258118+0000","last_unstale":"2024-07-26T23:58:50.825199+0000","last_undegraded":"2024-07-26T23:58:50.825199+0000","last_fullsized":"2024-07-26T23:58:50.825199+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:46:49.065664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1872,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.703259+0000","last_change":"2024-07-26T23:35:22.236663+0000","last_active":"2024-07-26T23:58:49.703259+0000","last_peered":"2024-07-26T23:58:49.703259+0000","last_clean":"2024-07-26T23:58:49.703259+0000","last_became_active":"2024-07-26T23:35:18.556903+0000","last_became_peered":"2024-07-26T23:35:18.556903+0000","last_unstale":"2024-07-26T23:58:49.703259+0000","last_undegraded":"2024-07-26T23:58:49.703259+0000","last_fullsized":"2024-07-26T23:58:49.703259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:27.775888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064553026999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1848,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.719879+0000","last_change":"2024-07-26T23:36:28.958153+0000","last_active":"2024-07-26T23:58:49.719879+0000","last_peered":"2024-07-26T23:58:49.719879+0000","last_clean":"2024-07-26T23:58:49.719879+0000","last_became_active":"2024-07-26T23:36:28.821361+0000","last_became_peered":"2024-07-26T23:36:28.821361+0000","last_unstale":"2024-07-26T23:58:49.719879+0000","last_undegraded":"2024-07-26T23:58:49.719879+0000","last_fullsized":"2024-07-26T23:58:49.719879+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:05:13.869444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.427603+0000","last_change":"2024-07-26T23:36:29.506650+0000","last_active":"2024-07-26T23:58:49.427603+0000","last_peered":"2024-07-26T23:58:49.427603+0000","last_clean":"2024-07-26T23:58:49.427603+0000","last_became_active":"2024-07-26T23:36:29.496365+0000","last_became_peered":"2024-07-26T23:36:29.496365+0000","last_unstale":"2024-07-26T23:58:49.427603+0000","last_undegraded":"2024-07-26T23:58:49.427603+0000","last_fullsized":"2024-07-26T23:58:49.427603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:15:34.642884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.011318+0000","last_change":"2024-07-26T23:36:33.306681+0000","last_active":"2024-07-26T23:58:48.011318+0000","last_peered":"2024-07-26T23:58:48.011318+0000","last_clean":"2024-07-26T23:58:48.011318+0000","last_became_active":"2024-07-26T23:36:33.301227+0000","last_became_peered":"2024-07-26T23:36:33.301227+0000","last_unstale":"2024-07-26T23:58:48.011318+0000","last_undegraded":"2024-07-26T23:58:48.011318+0000","last_fullsized":"2024-07-26T23:58:48.011318+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:47:53.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:47.996084+0000","last_change":"2024-07-26T23:36:30.287713+0000","last_active":"2024-07-26T23:58:47.996084+0000","last_peered":"2024-07-26T23:58:47.996084+0000","last_clean":"2024-07-26T23:58:47.996084+0000","last_became_active":"2024-07-26T23:36:30.280064+0000","last_became_peered":"2024-07-26T23:36:30.280064+0000","last_unstale":"2024-07-26T23:58:47.996084+0000","last_undegraded":"2024-07-26T23:58:47.996084+0000","last_fullsized":"2024-07-26T23:58:47.996084+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:43:33.794147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1838,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.211236+0000","last_change":"2024-07-26T23:36:29.655104+0000","last_active":"2024-07-26T23:58:48.211236+0000","last_peered":"2024-07-26T23:58:48.211236+0000","last_clean":"2024-07-26T23:58:48.211236+0000","last_became_active":"2024-07-26T23:36:29.640586+0000","last_became_peered":"2024-07-26T23:36:29.640586+0000","last_unstale":"2024-07-26T23:58:48.211236+0000","last_undegraded":"2024-07-26T23:58:48.211236+0000","last_fullsized":"2024-07-26T23:58:48.211236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:04:00.672728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.441671+0000","last_change":"2024-07-26T23:36:28.901469+0000","last_active":"2024-07-26T23:58:49.441671+0000","last_peered":"2024-07-26T23:58:49.441671+0000","last_clean":"2024-07-26T23:58:49.441671+0000","last_became_active":"2024-07-26T23:36:28.852843+0000","last_became_peered":"2024-07-26T23:36:28.852843+0000","last_unstale":"2024-07-26T23:58:49.441671+0000","last_undegraded":"2024-07-26T23:58:49.441671+0000","last_fullsized":"2024-07-26T23:58:49.441671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:09.082990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1839,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.176565+0000","last_change":"2024-07-26T23:36:31.130573+0000","last_active":"2024-07-26T23:58:48.176565+0000","last_peered":"2024-07-26T23:58:48.176565+0000","last_clean":"2024-07-26T23:58:48.176565+0000","last_became_active":"2024-07-26T23:36:31.124146+0000","last_became_peered":"2024-07-26T23:36:31.124146+0000","last_unstale":"2024-07-26T23:58:48.176565+0000","last_undegraded":"2024-07-26T23:58:48.176565+0000","last_fullsized":"2024-07-26T23:58:48.176565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:07:28.551245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1849,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.722616+0000","last_change":"2024-07-26T23:36:30.187288+0000","last_active":"2024-07-26T23:58:49.722616+0000","last_peered":"2024-07-26T23:58:49.722616+0000","last_clean":"2024-07-26T23:58:49.722616+0000","last_became_active":"2024-07-26T23:36:30.173625+0000","last_became_peered":"2024-07-26T23:36:30.173625+0000","last_unstale":"2024-07-26T23:58:49.722616+0000","last_undegraded":"2024-07-26T23:58:49.722616+0000","last_fullsized":"2024-07-26T23:58:49.722616+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:55:38.891400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1851,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.119799+0000","last_change":"2024-07-26T23:36:31.996757+0000","last_active":"2024-07-26T23:58:46.119799+0000","last_peered":"2024-07-26T23:58:46.119799+0000","last_clean":"2024-07-26T23:58:46.119799+0000","last_became_active":"2024-07-26T23:36:31.990984+0000","last_became_peered":"2024-07-26T23:36:31.990984+0000","last_unstale":"2024-07-26T23:58:46.119799+0000","last_undegraded":"2024-07-26T23:58:46.119799+0000","last_fullsized":"2024-07-26T23:58:46.119799+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:24:44.872192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1870,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.698237+0000","last_change":"2024-07-26T23:36:33.389570+0000","last_active":"2024-07-26T23:58:49.698237+0000","last_peered":"2024-07-26T23:58:49.698237+0000","last_clean":"2024-07-26T23:58:49.698237+0000","last_became_active":"2024-07-26T23:36:33.382005+0000","last_became_peered":"2024-07-26T23:36:33.382005+0000","last_unstale":"2024-07-26T23:58:49.698237+0000","last_undegraded":"2024-07-26T23:58:49.698237+0000","last_fullsized":"2024-07-26T23:58:49.698237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:33:51.810118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.430486+0000","last_change":"2024-07-26T23:36:28.719222+0000","last_active":"2024-07-26T23:58:49.430486+0000","last_peered":"2024-07-26T23:58:49.430486+0000","last_clean":"2024-07-26T23:58:49.430486+0000","last_became_active":"2024-07-26T23:36:28.654711+0000","last_became_peered":"2024-07-26T23:36:28.654711+0000","last_unstale":"2024-07-26T23:58:49.430486+0000","last_undegraded":"2024-07-26T23:58:49.430486+0000","last_fullsized":"2024-07-26T23:58:49.430486+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:06.380712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1869,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.819532+0000","last_change":"2024-07-26T23:36:36.812820+0000","last_active":"2024-07-26T23:58:50.819532+0000","last_peered":"2024-07-26T23:58:50.819532+0000","last_clean":"2024-07-26T23:58:50.819532+0000","last_became_active":"2024-07-26T23:36:36.803418+0000","last_became_peered":"2024-07-26T23:36:36.803418+0000","last_unstale":"2024-07-26T23:58:50.819532+0000","last_undegraded":"2024-07-26T23:58:50.819532+0000","last_fullsized":"2024-07-26T23:58:50.819532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:26:11.430116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":21,"seq":90194313503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94716,"kb_used_data":14952,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94277124,"statfs":{"total":96636764160,"available":96539774976,"internally_reserved":0,"allocated":15310848,"data_stored":9567401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313507,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90580,"kb_used_data":14964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281260,"statfs":{"total":96636764160,"available":96544010240,"internally_reserved":0,"allocated":15323136,"data_stored":9574126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[]},{"osd":3,"up_from":20,"seq":85899346208,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90584,"kb_used_data":14968,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281256,"statfs":{"total":96636764160,"available":96544006144,"internally_reserved":0,"allocated":15327232,"data_stored":9574145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346211,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91020,"kb_used_data":15404,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94280820,"statfs":{"total":96636764160,"available":96543559680,"internally_reserved":0,"allocated":15773696,"data_stored":10031266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411616,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82248,"kb_used_data":14964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94289592,"statfs":{"total":96636764160,"available":96552542208,"internally_reserved":0,"allocated":15323136,"data_stored":9574126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509728,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90672,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281168,"statfs":{"total":96636764160,"available":96543916032,"internally_reserved":0,"allocated":15429632,"data_stored":9694861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":110,"apply_latency_ms":110,"commit_latency_ns":110000000,"apply_latency_ns":110000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575135,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78096,"kb_used_data":14972,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94293744,"statfs":{"total":96636764160,"available":96556793856,"internally_reserved":0,"allocated":15331328,"data_stored":9574150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575133,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95140,"kb_used_data":15352,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94276700,"statfs":{"total":96636764160,"available":96539340800,"internally_reserved":0,"allocated":15720448,"data_stored":9990675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T23:58:52.773 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T23:58:52.773 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T23:58:52.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.858+0000 7f270a9cd640 10 monclient: get_monmap_and_config 2024-07-26T23:58:52.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.858+0000 7f270a9cd640 10 monclient: build_initial_monmap 2024-07-26T23:58:52.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.858+0000 7f270a9cd640 10 monclient: monmap: 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:52.859502+0000 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:52.859502+0000 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:52.874 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:52.875 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.876 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 1 Processor -- start 2024-07-26T23:58:52.876 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 1 -- start start 2024-07-26T23:58:52.876 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient: init 2024-07-26T23:58:52.876 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f2704141660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient(hunting): picked mon.noname-b con 0x7f270406d100 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27040af140 addr v1:172.21.15.139:6789/0 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27040ac140 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27040ae800 con 0x7f27040af140 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270413f1a0 con 0x7f27040ac140 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f27040ab9d0 con 0x7f270406d100 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.859+0000 7f270a9cd640 10 monclient(hunting): authenticate will time out at 2765.895863s 2024-07-26T23:58:52.877 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.860+0000 7f27037fe640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f27040af140 0x7f2704074060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:44736/0 (socket says 172.21.15.139:44736) 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.860+0000 7f27037fe640 1 -- 172.21.15.139:0/3502059340 learned_addr learned my addr 172.21.15.139:0/3502059340 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.860+0000 7f2703fff640 1 --2- 172.21.15.139:0/3502059340 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f2704141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.860+0000 7f2703fff640 10 monclient(hunting): get_auth_request con 0x7f270406d100 auth_method 0 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.860+0000 7f2703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.860+0000 7f2703fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.860+0000 7f2703fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.861+0000 7f2703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.861+0000 7f2703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.878 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.861+0000 7f2703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.879 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.862+0000 7f2702ffd640 1 -- 172.21.15.139:0/3502059340 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3902555388 0 0) 0x7f270413f1a0 con 0x7f27040ac140 2024-07-26T23:58:52.879 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.862+0000 7f2702ffd640 10 monclient(hunting): _negotiate 2024-07-26T23:58:52.879 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.862+0000 7f2702ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.862+0000 7f2702ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.862+0000 7f2702ffd640 10 monclient(hunting): my global_id is 20260 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.862+0000 7f2702ffd640 1 -- 172.21.15.139:0/3502059340 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26ec003520 con 0x7f27040ac140 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 10 monclient(hunting): handle_auth_done global_id 16811 payload 274 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 1 -- 172.21.15.139:0/3502059340 >> v1:172.21.15.177:6789/0 conn(0x7f27040ac140 legacy=0x7f270414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 1 -- 172.21.15.139:0/3502059340 >> v1:172.21.15.139:6789/0 conn(0x7f27040af140 legacy=0x7f2704074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 1 monclient: found mon.noname-b 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:58:52.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.863+0000 7f2703fff640 1 -- 172.21.15.139:0/3502059340 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27040abc20 con 0x7f270406d100 2024-07-26T23:58:52.881 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.865+0000 7f2703fff640 1 --2- 172.21.15.139:0/3502059340 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f2704141660 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f26f40097d0 tx=0x7f26f402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e138e29eb66c5679 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 1 -- 172.21.15.139:0/3502059340 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4035020 con 0x7f270406d100 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 10 monclient: dump: 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 10 monclient: _finish_auth 0 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f2702ffd640 1 -- 172.21.15.139:0/3502059340 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f4031800 con 0x7f270406d100 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.868+0000 7f270a9cd640 5 monclient: authenticate success, global_id 16811 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f2702ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f2702ffd640 1 -- 172.21.15.139:0/3502059340 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4031bd0 con 0x7f270406d100 2024-07-26T23:58:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 4 set_mon_vals no callback set 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f2702ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f2702ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f2702ffd640 10 monclient: dump: 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.887 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 10 monclient: shutdown 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 1 -- 172.21.15.139:0/3502059340 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 msgr2=0x7f2704141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 1 --2- 172.21.15.139:0/3502059340 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f2704141660 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f26f40097d0 tx=0x7f26f402b8a0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 1 -- 172.21.15.139:0/3502059340 shutdown_connections 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 1 --2- 172.21.15.139:0/3502059340 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f2704141660 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 1 -- 172.21.15.139:0/3502059340 >> 172.21.15.139:0/3502059340 conn(0x7f27040b28b0 msgr2=0x7f270406c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 1 -- 172.21.15.139:0/3502059340 shutdown_connections 2024-07-26T23:58:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.869+0000 7f270a9cd640 1 -- 172.21.15.139:0/3502059340 wait complete. 2024-07-26T23:58:52.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.870+0000 7f270a9cd640 10 monclient: build_initial_monmap 2024-07-26T23:58:52.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.870+0000 7f270a9cd640 10 monclient: monmap: 2024-07-26T23:58:52.892 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:52.892 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:52.892 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:52.892 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:52.892 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:52.892 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.892 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 librados: starting msgr at 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 librados: starting objecter 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 Processor -- start 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 -- start start 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 librados: setting wanted keys 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 librados: calling monclient init 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 10 monclient: init 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:58:52.893 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f270413e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 10 monclient(hunting): picked mon.noname-c con 0x7f270406d100 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27040ac140 0x7f2704137500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.871+0000 7f270a9cd640 10 monclient(hunting): picked mon.noname-b con 0x7f27040ac140 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27040af140 0x7f270413bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 10 monclient(hunting): picked mon.noname-a con 0x7f27040af140 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f2704074770 con 0x7f27040af140 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2704141db0 con 0x7f27040ac140 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:52.894 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f27040748f0 con 0x7f270406d100 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f270a9cd640 10 monclient(hunting): authenticate will time out at 2765.908034s 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2703fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f270413e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2708f6a640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27040af140 0x7f270413bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f27037fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27040ac140 0x7f2704137500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2703fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f270413e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48358/0 (socket says 172.21.15.139:48358) 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2703fff640 1 -- 172.21.15.139:0/4084477051 learned_addr learned my addr 172.21.15.139:0/4084477051 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2703fff640 10 monclient(hunting): get_auth_request con 0x7f270406d100 auth_method 0 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2703fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f2703fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f27037fe640 10 monclient(hunting): get_auth_request con 0x7f27040ac140 auth_method 0 2024-07-26T23:58:52.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f27037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f27037fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.872+0000 7f27037fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2708f6a640 10 monclient(hunting): get_auth_request con 0x7f27040af140 auth_method 0 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2708f6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2708f6a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2708f6a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f27037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f27037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f27037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2708f6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:52.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2708f6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.873+0000 7f2708f6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 10 monclient(hunting): handle_auth_done global_id 20263 payload 824 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 1 -- 172.21.15.139:0/4084477051 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 msgr2=0x7f270413e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f270413e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 1 -- 172.21.15.139:0/4084477051 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27040af140 msgr2=0x7f270413bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27040af140 0x7f270413bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 10 monclient: _finish_hunting 0 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 1 monclient: found mon.noname-b 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:52.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.876+0000 7f27037fe640 1 -- 172.21.15.139:0/4084477051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27041cc2b0 con 0x7f27040ac140 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.878+0000 7f27037fe640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27040ac140 0x7f2704137500 secure :-1 s=READY pgs=4927 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c310 tx=0x7f26f800c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.878+0000 7f2700ff9640 10 client.?.objecter ms_handle_connect 0x7f27040ac140 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.878+0000 7f2700ff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 1 -- 172.21.15.139:0/4084477051 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8017070 con 0x7f27040ac140 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: dump: 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.899 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: _finish_auth 0 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: _check_auth_tickets 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 1 -- 172.21.15.139:0/4084477051 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f8005ce0 con 0x7f27040ac140 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 5 monclient: authenticate success, global_id 20263 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 10 monclient: _renew_subs 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27041cc590 con 0x7f27040ac140 2024-07-26T23:58:52.900 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 1 -- 172.21.15.139:0/4084477051 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8007310 con 0x7f27040ac140 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f2700ff9640 10 monclient: dump: 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:52.901 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 10 client.20263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 10 monclient: _renew_subs 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f27027fc640 4 set_mon_vals no callback set 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27041cb300 con 0x7f27040ac140 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.882+0000 7f270a9cd640 1 librados: init done 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.883+0000 7f270a9cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.883+0000 7f270a9cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:52.902 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.883+0000 7f270a9cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:52.903 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.883+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270413e9e0 con 0x7f27040ac140 2024-07-26T23:58:52.919 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.904+0000 7f2700ff9640 1 -- 172.21.15.139:0/4084477051 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f26f801a070 con 0x7f27040ac140 2024-07-26T23:58:52.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.905+0000 7f2700ff9640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f26f00789f0 0x7f26f007aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:52.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.905+0000 7f2703fff640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f26f00789f0 0x7f26f007aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:52.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.905+0000 7f2703fff640 10 monclient: get_auth_request con 0x7f26f00789f0 auth_method 0 2024-07-26T23:58:52.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.905+0000 7f2703fff640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f26f00789f0 0x7f26f007aeb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f26f402b8e0 tx=0x7f26f4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:52.920 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:52.905+0000 7f2700ff9640 10 client.20263.objecter ms_handle_connect 0x7f26f00789f0 2024-07-26T23:58:53.148 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.133+0000 7f2700ff9640 1 -- 172.21.15.139:0/4084477051 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(969..969 src has 501..969) ==== 6365+0+0 (secure 0 0 0) 0x7f26f805cc80 con 0x7f27040ac140 2024-07-26T23:58:53.148 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.133+0000 7f2700ff9640 10 client.20263.objecter ms_dispatch 0x7f2704000b90 osd_map(969..969 src has 501..969) 2024-07-26T23:58:53.148 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.133+0000 7f2700ff9640 3 client.20263.objecter handle_osd_map got epochs [969,969] > 0 2024-07-26T23:58:53.148 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.133+0000 7f2700ff9640 3 client.20263.objecter handle_osd_map decoding full epoch 969 2024-07-26T23:58:53.148 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.133+0000 7f2700ff9640 20 client.20263.objecter dump_active .. 0 homeless 2024-07-26T23:58:53.158 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.143+0000 7f2700ff9640 1 -- 172.21.15.139:0/4084477051 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f26f8021020 con 0x7f27040ac140 2024-07-26T23:58:53.158 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.143+0000 7f2700ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:53.159 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.143+0000 7f2700ff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:53.352 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.337+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2704141660 con 0x7f26f00789f0 2024-07-26T23:58:53.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.339+0000 7f2700ff9640 1 -- 172.21.15.139:0/4084477051 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116459 (secure 0 0 0) 0x7f2704141660 con 0x7f26f00789f0 2024-07-26T23:58:53.355 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:58:53.356 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-07-26T23:58:53.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.341+0000 7f270a9cd640 10 librados: watch_flush enter 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.341+0000 7f270a9cd640 10 librados: watch_flush exit 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.341+0000 7f270a9cd640 20 client.20263.objecter shutdown clearing up homeless session... 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.341+0000 7f270a9cd640 10 client.20263.objecter successfully canceled tick 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f26f00789f0 msgr2=0x7f26f007aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f26f00789f0 0x7f26f007aeb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f26f402b8e0 tx=0x7f26f4008030 comp rx=0 tx=0).stop 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 10 monclient: shutdown 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27040ac140 msgr2=0x7f2704137500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27040ac140 0x7f2704137500 secure :-1 s=READY pgs=4927 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c310 tx=0x7f26f800c7d0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 shutdown_connections 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f26f00789f0 0x7f26f007aeb0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27040af140 0x7f270413bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27040ac140 0x7f2704137500 unknown :-1 s=CLOSED pgs=4927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 --2- 172.21.15.139:0/4084477051 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f270406d100 0x7f270413e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 >> 172.21.15.139:0/4084477051 conn(0x7f27040b28b0 msgr2=0x7f2704074040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:53.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 shutdown_connections 2024-07-26T23:58:53.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 -- 172.21.15.139:0/4084477051 wait complete. 2024-07-26T23:58:53.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.342+0000 7f270a9cd640 1 librados: shutdown 2024-07-26T23:58:53.369 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":1688,"stamp":"2024-07-26T23:58:52.216184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":52,"num_read_kb":43,"num_write":66,"num_write_kb":593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":713056,"kb_used_data":120644,"kb_used_omap":12,"kb_used_meta":592179,"kb_avail":754261664,"statfs":{"total":773094113280,"available":772363943936,"internally_reserved":0,"allocated":123539456,"data_stored":77580750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":606391856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":432,"apply_latency_ms":432,"commit_latency_ns":432000000,"apply_latency_ns":432000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.004226"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1850,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.132348+0000","last_change":"2024-07-26T23:36:37.015743+0000","last_active":"2024-07-26T23:58:46.132348+0000","last_peered":"2024-07-26T23:58:46.132348+0000","last_clean":"2024-07-26T23:58:46.132348+0000","last_became_active":"2024-07-26T23:36:37.010177+0000","last_became_peered":"2024-07-26T23:36:37.010177+0000","last_unstale":"2024-07-26T23:58:46.132348+0000","last_undegraded":"2024-07-26T23:58:46.132348+0000","last_fullsized":"2024-07-26T23:58:46.132348+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:54:54.240713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1868,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.841031+0000","last_change":"2024-07-26T23:36:32.386250+0000","last_active":"2024-07-26T23:58:50.841031+0000","last_peered":"2024-07-26T23:58:50.841031+0000","last_clean":"2024-07-26T23:58:50.841031+0000","last_became_active":"2024-07-26T23:36:32.380680+0000","last_became_peered":"2024-07-26T23:36:32.380680+0000","last_unstale":"2024-07-26T23:58:50.841031+0000","last_undegraded":"2024-07-26T23:58:50.841031+0000","last_fullsized":"2024-07-26T23:58:50.841031+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:19:53.312577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1846,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.439029+0000","last_change":"2024-07-26T23:36:32.478927+0000","last_active":"2024-07-26T23:58:49.439029+0000","last_peered":"2024-07-26T23:58:49.439029+0000","last_clean":"2024-07-26T23:58:49.439029+0000","last_became_active":"2024-07-26T23:36:32.473374+0000","last_became_peered":"2024-07-26T23:36:32.473374+0000","last_unstale":"2024-07-26T23:58:49.439029+0000","last_undegraded":"2024-07-26T23:58:49.439029+0000","last_fullsized":"2024-07-26T23:58:49.439029+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:13:31.972041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.701385+0000","last_change":"2024-07-26T23:36:45.735883+0000","last_active":"2024-07-26T23:58:49.701385+0000","last_peered":"2024-07-26T23:58:49.701385+0000","last_clean":"2024-07-26T23:58:49.701385+0000","last_became_active":"2024-07-26T23:36:45.718574+0000","last_became_peered":"2024-07-26T23:36:45.718574+0000","last_unstale":"2024-07-26T23:58:49.701385+0000","last_undegraded":"2024-07-26T23:58:49.701385+0000","last_fullsized":"2024-07-26T23:58:49.701385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:22:53.643566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1851,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.711009+0000","last_change":"2024-07-26T23:36:33.937849+0000","last_active":"2024-07-26T23:58:49.711009+0000","last_peered":"2024-07-26T23:58:49.711009+0000","last_clean":"2024-07-26T23:58:49.711009+0000","last_became_active":"2024-07-26T23:36:33.915707+0000","last_became_peered":"2024-07-26T23:36:33.915707+0000","last_unstale":"2024-07-26T23:58:49.711009+0000","last_undegraded":"2024-07-26T23:58:49.711009+0000","last_fullsized":"2024-07-26T23:58:49.711009+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.705473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"44'7","reported_seq":1854,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.005889+0000","last_change":"2024-07-26T23:36:30.524945+0000","last_active":"2024-07-26T23:58:48.005889+0000","last_peered":"2024-07-26T23:58:48.005889+0000","last_clean":"2024-07-26T23:58:48.005889+0000","last_became_active":"2024-07-26T23:36:30.507651+0000","last_became_peered":"2024-07-26T23:36:30.507651+0000","last_unstale":"2024-07-26T23:58:48.005889+0000","last_undegraded":"2024-07-26T23:58:48.005889+0000","last_fullsized":"2024-07-26T23:58:48.005889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:36:56.132284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1845,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.428542+0000","last_change":"2024-07-26T23:36:29.482671+0000","last_active":"2024-07-26T23:58:49.428542+0000","last_peered":"2024-07-26T23:58:49.428542+0000","last_clean":"2024-07-26T23:58:49.428542+0000","last_became_active":"2024-07-26T23:36:29.468516+0000","last_became_peered":"2024-07-26T23:36:29.468516+0000","last_unstale":"2024-07-26T23:58:49.428542+0000","last_undegraded":"2024-07-26T23:58:49.428542+0000","last_fullsized":"2024-07-26T23:58:49.428542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:10:56.313880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1871,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.705157+0000","last_change":"2024-07-26T23:36:50.102068+0000","last_active":"2024-07-26T23:58:49.705157+0000","last_peered":"2024-07-26T23:58:49.705157+0000","last_clean":"2024-07-26T23:58:49.705157+0000","last_became_active":"2024-07-26T23:36:50.085394+0000","last_became_peered":"2024-07-26T23:36:50.085394+0000","last_unstale":"2024-07-26T23:58:49.705157+0000","last_undegraded":"2024-07-26T23:58:49.705157+0000","last_fullsized":"2024-07-26T23:58:49.705157+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:03:21.264586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1845,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.436927+0000","last_change":"2024-07-26T23:36:32.322928+0000","last_active":"2024-07-26T23:58:49.436927+0000","last_peered":"2024-07-26T23:58:49.436927+0000","last_clean":"2024-07-26T23:58:49.436927+0000","last_became_active":"2024-07-26T23:36:32.315924+0000","last_became_peered":"2024-07-26T23:36:32.315924+0000","last_unstale":"2024-07-26T23:58:49.436927+0000","last_undegraded":"2024-07-26T23:58:49.436927+0000","last_fullsized":"2024-07-26T23:58:49.436927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:29:47.697657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.737182+0000","last_change":"2024-07-26T23:36:33.078206+0000","last_active":"2024-07-26T23:58:49.737182+0000","last_peered":"2024-07-26T23:58:49.737182+0000","last_clean":"2024-07-26T23:58:49.737182+0000","last_became_active":"2024-07-26T23:36:33.069066+0000","last_became_peered":"2024-07-26T23:36:33.069066+0000","last_unstale":"2024-07-26T23:58:49.737182+0000","last_undegraded":"2024-07-26T23:58:49.737182+0000","last_fullsized":"2024-07-26T23:58:49.737182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:58:53.548672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1870,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.846114+0000","last_change":"2024-07-26T23:36:32.994273+0000","last_active":"2024-07-26T23:58:50.846114+0000","last_peered":"2024-07-26T23:58:50.846114+0000","last_clean":"2024-07-26T23:58:50.846114+0000","last_became_active":"2024-07-26T23:36:32.984857+0000","last_became_peered":"2024-07-26T23:36:32.984857+0000","last_unstale":"2024-07-26T23:58:50.846114+0000","last_undegraded":"2024-07-26T23:58:50.846114+0000","last_fullsized":"2024-07-26T23:58:50.846114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:54:04.890736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.141761+0000","last_change":"2024-07-26T23:36:30.566094+0000","last_active":"2024-07-26T23:58:48.141761+0000","last_peered":"2024-07-26T23:58:48.141761+0000","last_clean":"2024-07-26T23:58:48.141761+0000","last_became_active":"2024-07-26T23:36:30.560131+0000","last_became_peered":"2024-07-26T23:36:30.560131+0000","last_unstale":"2024-07-26T23:58:48.141761+0000","last_undegraded":"2024-07-26T23:58:48.141761+0000","last_fullsized":"2024-07-26T23:58:48.141761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:58:36.316935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1850,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.695745+0000","last_change":"2024-07-26T23:36:44.212854+0000","last_active":"2024-07-26T23:58:49.695745+0000","last_peered":"2024-07-26T23:58:49.695745+0000","last_clean":"2024-07-26T23:58:49.695745+0000","last_became_active":"2024-07-26T23:36:44.149525+0000","last_became_peered":"2024-07-26T23:36:44.149525+0000","last_unstale":"2024-07-26T23:58:49.695745+0000","last_undegraded":"2024-07-26T23:58:49.695745+0000","last_fullsized":"2024-07-26T23:58:49.695745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:09:35.012505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'32","reported_seq":1866,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.053298+0000","last_change":"2024-07-26T23:35:17.404909+0000","last_active":"2024-07-26T23:58:48.053298+0000","last_peered":"2024-07-26T23:58:48.053298+0000","last_clean":"2024-07-26T23:58:48.053298+0000","last_became_active":"2024-07-26T23:35:16.707863+0000","last_became_peered":"2024-07-26T23:35:16.707863+0000","last_unstale":"2024-07-26T23:58:48.053298+0000","last_undegraded":"2024-07-26T23:58:48.053298+0000","last_fullsized":"2024-07-26T23:58:48.053298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:01.337925+0000","last_clean_scrub_stamp":"2024-07-26T23:35:01.337925+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:50:39.235610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":1877,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.688571+0000","last_change":"2024-07-26T23:35:22.241392+0000","last_active":"2024-07-26T23:58:49.688571+0000","last_peered":"2024-07-26T23:58:49.688571+0000","last_clean":"2024-07-26T23:58:49.688571+0000","last_became_active":"2024-07-26T23:35:19.087174+0000","last_became_peered":"2024-07-26T23:35:19.087174+0000","last_unstale":"2024-07-26T23:58:49.688571+0000","last_undegraded":"2024-07-26T23:58:49.688571+0000","last_fullsized":"2024-07-26T23:58:49.688571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:54:22.975250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.074002362000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1855,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.060718+0000","last_change":"2024-07-26T23:36:39.448493+0000","last_active":"2024-07-26T23:58:48.060718+0000","last_peered":"2024-07-26T23:58:48.060718+0000","last_clean":"2024-07-26T23:58:48.060718+0000","last_became_active":"2024-07-26T23:36:39.441853+0000","last_became_peered":"2024-07-26T23:36:39.441853+0000","last_unstale":"2024-07-26T23:58:48.060718+0000","last_undegraded":"2024-07-26T23:58:48.060718+0000","last_fullsized":"2024-07-26T23:58:48.060718+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:33:54.532094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1876,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.682140+0000","last_change":"2024-07-26T23:35:22.212052+0000","last_active":"2024-07-26T23:58:49.682140+0000","last_peered":"2024-07-26T23:58:49.682140+0000","last_clean":"2024-07-26T23:58:49.682140+0000","last_became_active":"2024-07-26T23:35:17.846505+0000","last_became_peered":"2024-07-26T23:35:17.846505+0000","last_unstale":"2024-07-26T23:58:49.682140+0000","last_undegraded":"2024-07-26T23:58:49.682140+0000","last_fullsized":"2024-07-26T23:58:49.682140+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:31.369623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11625252899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1848,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.705039+0000","last_change":"2024-07-26T23:36:33.996623+0000","last_active":"2024-07-26T23:58:49.705039+0000","last_peered":"2024-07-26T23:58:49.705039+0000","last_clean":"2024-07-26T23:58:49.705039+0000","last_became_active":"2024-07-26T23:36:33.958339+0000","last_became_peered":"2024-07-26T23:36:33.958339+0000","last_unstale":"2024-07-26T23:58:49.705039+0000","last_undegraded":"2024-07-26T23:58:49.705039+0000","last_fullsized":"2024-07-26T23:58:49.705039+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:19.336284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1880,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.013967+0000","last_change":"2024-07-26T23:35:22.093863+0000","last_active":"2024-07-26T23:58:48.013967+0000","last_peered":"2024-07-26T23:58:48.013967+0000","last_clean":"2024-07-26T23:58:48.013967+0000","last_became_active":"2024-07-26T23:35:17.814359+0000","last_became_peered":"2024-07-26T23:35:17.814359+0000","last_unstale":"2024-07-26T23:58:48.013967+0000","last_undegraded":"2024-07-26T23:58:48.013967+0000","last_fullsized":"2024-07-26T23:58:48.013967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:27:36.663020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060284956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.050550+0000","last_change":"2024-07-26T23:36:32.118711+0000","last_active":"2024-07-26T23:58:48.050550+0000","last_peered":"2024-07-26T23:58:48.050550+0000","last_clean":"2024-07-26T23:58:48.050550+0000","last_became_active":"2024-07-26T23:36:32.112540+0000","last_became_peered":"2024-07-26T23:36:32.112540+0000","last_unstale":"2024-07-26T23:58:48.050550+0000","last_undegraded":"2024-07-26T23:58:48.050550+0000","last_fullsized":"2024-07-26T23:58:48.050550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:32:07.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":1878,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.692788+0000","last_change":"2024-07-26T23:35:22.220704+0000","last_active":"2024-07-26T23:58:49.692788+0000","last_peered":"2024-07-26T23:58:49.692788+0000","last_clean":"2024-07-26T23:58:49.692788+0000","last_became_active":"2024-07-26T23:35:18.492082+0000","last_became_peered":"2024-07-26T23:35:18.492082+0000","last_unstale":"2024-07-26T23:58:49.692788+0000","last_undegraded":"2024-07-26T23:58:49.692788+0000","last_fullsized":"2024-07-26T23:58:49.692788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:00.865495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098681633000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1871,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.700871+0000","last_change":"2024-07-26T23:36:29.379730+0000","last_active":"2024-07-26T23:58:49.700871+0000","last_peered":"2024-07-26T23:58:49.700871+0000","last_clean":"2024-07-26T23:58:49.700871+0000","last_became_active":"2024-07-26T23:36:29.374355+0000","last_became_peered":"2024-07-26T23:36:29.374355+0000","last_unstale":"2024-07-26T23:58:49.700871+0000","last_undegraded":"2024-07-26T23:58:49.700871+0000","last_fullsized":"2024-07-26T23:58:49.700871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:55:50.796408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1875,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.135256+0000","last_change":"2024-07-26T23:35:22.376891+0000","last_active":"2024-07-26T23:58:46.135256+0000","last_peered":"2024-07-26T23:58:46.135256+0000","last_clean":"2024-07-26T23:58:46.135256+0000","last_became_active":"2024-07-26T23:35:18.507138+0000","last_became_peered":"2024-07-26T23:35:18.507138+0000","last_unstale":"2024-07-26T23:58:46.135256+0000","last_undegraded":"2024-07-26T23:58:46.135256+0000","last_fullsized":"2024-07-26T23:58:46.135256+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:56:22.382897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047472209000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.099229+0000","last_change":"2024-07-26T23:36:32.850160+0000","last_active":"2024-07-26T23:58:48.099229+0000","last_peered":"2024-07-26T23:58:48.099229+0000","last_clean":"2024-07-26T23:58:48.099229+0000","last_became_active":"2024-07-26T23:36:32.842079+0000","last_became_peered":"2024-07-26T23:36:32.842079+0000","last_unstale":"2024-07-26T23:58:48.099229+0000","last_undegraded":"2024-07-26T23:58:48.099229+0000","last_fullsized":"2024-07-26T23:58:48.099229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:53:03.317423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1875,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.697649+0000","last_change":"2024-07-26T23:35:22.251048+0000","last_active":"2024-07-26T23:58:49.697649+0000","last_peered":"2024-07-26T23:58:49.697649+0000","last_clean":"2024-07-26T23:58:49.697649+0000","last_became_active":"2024-07-26T23:35:17.925979+0000","last_became_peered":"2024-07-26T23:35:17.925979+0000","last_unstale":"2024-07-26T23:58:49.697649+0000","last_undegraded":"2024-07-26T23:58:49.697649+0000","last_fullsized":"2024-07-26T23:58:49.697649+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:20.031998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073846415999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1857,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:47.993381+0000","last_change":"2024-07-26T23:36:31.631363+0000","last_active":"2024-07-26T23:58:47.993381+0000","last_peered":"2024-07-26T23:58:47.993381+0000","last_clean":"2024-07-26T23:58:47.993381+0000","last_became_active":"2024-07-26T23:36:31.543107+0000","last_became_peered":"2024-07-26T23:36:31.543107+0000","last_unstale":"2024-07-26T23:58:47.993381+0000","last_undegraded":"2024-07-26T23:58:47.993381+0000","last_fullsized":"2024-07-26T23:58:47.993381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:49:57.654672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1892,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.763722+0000","last_change":"2024-07-26T23:35:22.979978+0000","last_active":"2024-07-26T23:58:49.763722+0000","last_peered":"2024-07-26T23:58:49.763722+0000","last_clean":"2024-07-26T23:58:49.763722+0000","last_became_active":"2024-07-26T23:35:18.587303+0000","last_became_peered":"2024-07-26T23:35:18.587303+0000","last_unstale":"2024-07-26T23:58:49.763722+0000","last_undegraded":"2024-07-26T23:58:49.763722+0000","last_fullsized":"2024-07-26T23:58:49.763722+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:51:04.403633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.077704690000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1870,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.825199+0000","last_change":"2024-07-26T23:36:50.276152+0000","last_active":"2024-07-26T23:58:50.825199+0000","last_peered":"2024-07-26T23:58:50.825199+0000","last_clean":"2024-07-26T23:58:50.825199+0000","last_became_active":"2024-07-26T23:36:50.258118+0000","last_became_peered":"2024-07-26T23:36:50.258118+0000","last_unstale":"2024-07-26T23:58:50.825199+0000","last_undegraded":"2024-07-26T23:58:50.825199+0000","last_fullsized":"2024-07-26T23:58:50.825199+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:46:49.065664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1872,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.703259+0000","last_change":"2024-07-26T23:35:22.236663+0000","last_active":"2024-07-26T23:58:49.703259+0000","last_peered":"2024-07-26T23:58:49.703259+0000","last_clean":"2024-07-26T23:58:49.703259+0000","last_became_active":"2024-07-26T23:35:18.556903+0000","last_became_peered":"2024-07-26T23:35:18.556903+0000","last_unstale":"2024-07-26T23:58:49.703259+0000","last_undegraded":"2024-07-26T23:58:49.703259+0000","last_fullsized":"2024-07-26T23:58:49.703259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:27.775888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064553026999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1848,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.719879+0000","last_change":"2024-07-26T23:36:28.958153+0000","last_active":"2024-07-26T23:58:49.719879+0000","last_peered":"2024-07-26T23:58:49.719879+0000","last_clean":"2024-07-26T23:58:49.719879+0000","last_became_active":"2024-07-26T23:36:28.821361+0000","last_became_peered":"2024-07-26T23:36:28.821361+0000","last_unstale":"2024-07-26T23:58:49.719879+0000","last_undegraded":"2024-07-26T23:58:49.719879+0000","last_fullsized":"2024-07-26T23:58:49.719879+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:05:13.869444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.427603+0000","last_change":"2024-07-26T23:36:29.506650+0000","last_active":"2024-07-26T23:58:49.427603+0000","last_peered":"2024-07-26T23:58:49.427603+0000","last_clean":"2024-07-26T23:58:49.427603+0000","last_became_active":"2024-07-26T23:36:29.496365+0000","last_became_peered":"2024-07-26T23:36:29.496365+0000","last_unstale":"2024-07-26T23:58:49.427603+0000","last_undegraded":"2024-07-26T23:58:49.427603+0000","last_fullsized":"2024-07-26T23:58:49.427603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:15:34.642884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.011318+0000","last_change":"2024-07-26T23:36:33.306681+0000","last_active":"2024-07-26T23:58:48.011318+0000","last_peered":"2024-07-26T23:58:48.011318+0000","last_clean":"2024-07-26T23:58:48.011318+0000","last_became_active":"2024-07-26T23:36:33.301227+0000","last_became_peered":"2024-07-26T23:36:33.301227+0000","last_unstale":"2024-07-26T23:58:48.011318+0000","last_undegraded":"2024-07-26T23:58:48.011318+0000","last_fullsized":"2024-07-26T23:58:48.011318+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:47:53.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1856,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:47.996084+0000","last_change":"2024-07-26T23:36:30.287713+0000","last_active":"2024-07-26T23:58:47.996084+0000","last_peered":"2024-07-26T23:58:47.996084+0000","last_clean":"2024-07-26T23:58:47.996084+0000","last_became_active":"2024-07-26T23:36:30.280064+0000","last_became_peered":"2024-07-26T23:36:30.280064+0000","last_unstale":"2024-07-26T23:58:47.996084+0000","last_undegraded":"2024-07-26T23:58:47.996084+0000","last_fullsized":"2024-07-26T23:58:47.996084+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:43:33.794147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1838,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.211236+0000","last_change":"2024-07-26T23:36:29.655104+0000","last_active":"2024-07-26T23:58:48.211236+0000","last_peered":"2024-07-26T23:58:48.211236+0000","last_clean":"2024-07-26T23:58:48.211236+0000","last_became_active":"2024-07-26T23:36:29.640586+0000","last_became_peered":"2024-07-26T23:36:29.640586+0000","last_unstale":"2024-07-26T23:58:48.211236+0000","last_undegraded":"2024-07-26T23:58:48.211236+0000","last_fullsized":"2024-07-26T23:58:48.211236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:04:00.672728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.441671+0000","last_change":"2024-07-26T23:36:28.901469+0000","last_active":"2024-07-26T23:58:49.441671+0000","last_peered":"2024-07-26T23:58:49.441671+0000","last_clean":"2024-07-26T23:58:49.441671+0000","last_became_active":"2024-07-26T23:36:28.852843+0000","last_became_peered":"2024-07-26T23:36:28.852843+0000","last_unstale":"2024-07-26T23:58:49.441671+0000","last_undegraded":"2024-07-26T23:58:49.441671+0000","last_fullsized":"2024-07-26T23:58:49.441671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:15:09.082990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1839,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-07-26T23:58:48.176565+0000","last_change":"2024-07-26T23:36:31.130573+0000","last_active":"2024-07-26T23:58:48.176565+0000","last_peered":"2024-07-26T23:58:48.176565+0000","last_clean":"2024-07-26T23:58:48.176565+0000","last_became_active":"2024-07-26T23:36:31.124146+0000","last_became_peered":"2024-07-26T23:36:31.124146+0000","last_unstale":"2024-07-26T23:58:48.176565+0000","last_undegraded":"2024-07-26T23:58:48.176565+0000","last_fullsized":"2024-07-26T23:58:48.176565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:07:28.551245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1849,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.722616+0000","last_change":"2024-07-26T23:36:30.187288+0000","last_active":"2024-07-26T23:58:49.722616+0000","last_peered":"2024-07-26T23:58:49.722616+0000","last_clean":"2024-07-26T23:58:49.722616+0000","last_became_active":"2024-07-26T23:36:30.173625+0000","last_became_peered":"2024-07-26T23:36:30.173625+0000","last_unstale":"2024-07-26T23:58:49.722616+0000","last_undegraded":"2024-07-26T23:58:49.722616+0000","last_fullsized":"2024-07-26T23:58:49.722616+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T05:55:38.891400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1851,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-07-26T23:58:46.119799+0000","last_change":"2024-07-26T23:36:31.996757+0000","last_active":"2024-07-26T23:58:46.119799+0000","last_peered":"2024-07-26T23:58:46.119799+0000","last_clean":"2024-07-26T23:58:46.119799+0000","last_became_active":"2024-07-26T23:36:31.990984+0000","last_became_peered":"2024-07-26T23:36:31.990984+0000","last_unstale":"2024-07-26T23:58:46.119799+0000","last_undegraded":"2024-07-26T23:58:46.119799+0000","last_fullsized":"2024-07-26T23:58:46.119799+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:24:44.872192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1870,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.698237+0000","last_change":"2024-07-26T23:36:33.389570+0000","last_active":"2024-07-26T23:58:49.698237+0000","last_peered":"2024-07-26T23:58:49.698237+0000","last_clean":"2024-07-26T23:58:49.698237+0000","last_became_active":"2024-07-26T23:36:33.382005+0000","last_became_peered":"2024-07-26T23:36:33.382005+0000","last_unstale":"2024-07-26T23:58:49.698237+0000","last_undegraded":"2024-07-26T23:58:49.698237+0000","last_fullsized":"2024-07-26T23:58:49.698237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:33:51.810118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1844,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-07-26T23:58:49.430486+0000","last_change":"2024-07-26T23:36:28.719222+0000","last_active":"2024-07-26T23:58:49.430486+0000","last_peered":"2024-07-26T23:58:49.430486+0000","last_clean":"2024-07-26T23:58:49.430486+0000","last_became_active":"2024-07-26T23:36:28.654711+0000","last_became_peered":"2024-07-26T23:36:28.654711+0000","last_unstale":"2024-07-26T23:58:49.430486+0000","last_undegraded":"2024-07-26T23:58:49.430486+0000","last_fullsized":"2024-07-26T23:58:49.430486+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:18:06.380712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1869,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-07-26T23:58:50.819532+0000","last_change":"2024-07-26T23:36:36.812820+0000","last_active":"2024-07-26T23:58:50.819532+0000","last_peered":"2024-07-26T23:58:50.819532+0000","last_clean":"2024-07-26T23:58:50.819532+0000","last_became_active":"2024-07-26T23:36:36.803418+0000","last_became_peered":"2024-07-26T23:36:36.803418+0000","last_unstale":"2024-07-26T23:58:50.819532+0000","last_undegraded":"2024-07-26T23:58:50.819532+0000","last_fullsized":"2024-07-26T23:58:50.819532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:26:11.430116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":21,"seq":90194313503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94716,"kb_used_data":14952,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94277124,"statfs":{"total":96636764160,"available":96539774976,"internally_reserved":0,"allocated":15310848,"data_stored":9567401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313507,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90580,"kb_used_data":14964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281260,"statfs":{"total":96636764160,"available":96544010240,"internally_reserved":0,"allocated":15323136,"data_stored":9574126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[]},{"osd":3,"up_from":20,"seq":85899346208,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90584,"kb_used_data":14968,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281256,"statfs":{"total":96636764160,"available":96544006144,"internally_reserved":0,"allocated":15327232,"data_stored":9574145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346211,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91020,"kb_used_data":15404,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94280820,"statfs":{"total":96636764160,"available":96543559680,"internally_reserved":0,"allocated":15773696,"data_stored":10031266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411616,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82248,"kb_used_data":14964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94289592,"statfs":{"total":96636764160,"available":96552542208,"internally_reserved":0,"allocated":15323136,"data_stored":9574126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509728,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90672,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281168,"statfs":{"total":96636764160,"available":96543916032,"internally_reserved":0,"allocated":15429632,"data_stored":9694861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":110,"apply_latency_ms":110,"commit_latency_ns":110000000,"apply_latency_ns":110000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575135,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78096,"kb_used_data":14972,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94293744,"statfs":{"total":96636764160,"available":96556793856,"internally_reserved":0,"allocated":15331328,"data_stored":9574150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575133,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95140,"kb_used_data":15352,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94276700,"statfs":{"total":96636764160,"available":96539340800,"internally_reserved":0,"allocated":15720448,"data_stored":9990675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T23:58:53.371 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T23:58:53.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.454+0000 7fe881aae640 10 monclient: get_monmap_and_config 2024-07-26T23:58:53.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.454+0000 7fe881aae640 10 monclient: build_initial_monmap 2024-07-26T23:58:53.469 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.454+0000 7fe881aae640 10 monclient: monmap: 2024-07-26T23:58:53.469 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:53.469 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:53.455259+0000 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:53.455259+0000 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:53.470 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:53.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.454+0000 7fe881aae640 1 Processor -- start 2024-07-26T23:58:53.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 1 -- start start 2024-07-26T23:58:53.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient: init 2024-07-26T23:58:53.473 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe87c1470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe87c0b41c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fe87c14d1e0 0x7fe87c14f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient(hunting): picked mon.noname-a con 0x7fe87c14d1e0 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87c147ce0 con 0x7fe87c1470c0 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87c0acaf0 con 0x7fe87c0b41c0 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fe87c1477c0 con 0x7fe87c14d1e0 2024-07-26T23:58:53.474 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.455+0000 7fe881aae640 10 monclient(hunting): authenticate will time out at 2766.491687s 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.456+0000 7fe87bfff640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fe87c14d1e0 0x7fe87c14f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.456+0000 7fe87affd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe87c0b41c0 0x7fe87c0b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:53910/0 (socket says 172.21.15.139:53910) 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.456+0000 7fe87affd640 1 -- 172.21.15.139:0/1023551667 learned_addr learned my addr 172.21.15.139:0/1023551667 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.456+0000 7fe87bfff640 10 monclient(hunting): get_auth_request con 0x7fe87c14d1e0 auth_method 0 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.456+0000 7fe87bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.456+0000 7fe87bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.456+0000 7fe87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:53.475 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.457+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:53.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.457+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:53.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.457+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:53.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.458+0000 7fe87a7fc640 1 -- 172.21.15.139:0/1023551667 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1695942884 0 0) 0x7fe87c0acaf0 con 0x7fe87c0b41c0 2024-07-26T23:58:53.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.458+0000 7fe87a7fc640 10 monclient(hunting): _negotiate 2024-07-26T23:58:53.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.458+0000 7fe87a7fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:53.476 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.458+0000 7fe87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:53.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.458+0000 7fe87a7fc640 10 monclient(hunting): my global_id is 20266 2024-07-26T23:58:53.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.458+0000 7fe87a7fc640 1 -- 172.21.15.139:0/1023551667 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe868003520 con 0x7fe87c0b41c0 2024-07-26T23:58:53.477 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 10 monclient(hunting): handle_auth_done global_id 15534 payload 274 2024-07-26T23:58:53.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 1 -- 172.21.15.139:0/1023551667 >> v1:172.21.15.177:6789/0 conn(0x7fe87c0b41c0 legacy=0x7fe87c0b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 1 -- 172.21.15.139:0/1023551667 >> v1:172.21.15.139:6789/0 conn(0x7fe87c1470c0 legacy=0x7fe87c140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:53.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 1 monclient: found mon.noname-a 2024-07-26T23:58:53.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:53.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:53.478 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.459+0000 7fe87bfff640 1 -- 172.21.15.139:0/1023551667 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c1474a0 con 0x7fe87c14d1e0 2024-07-26T23:58:53.480 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.465+0000 7fe87bfff640 1 --2- 172.21.15.139:0/1023551667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c14d1e0 0x7fe87c14f5d0 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0097a0 tx=0x7fe86c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e3226c164896cb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 1 -- 172.21.15.139:0/1023551667 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c035020 con 0x7fe87c14d1e0 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: dump: 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:53.484 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: _finish_auth 0 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: _check_auth_tickets 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 1 -- 172.21.15.139:0/1023551667 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c031830 con 0x7fe87c14d1e0 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe881aae640 5 monclient: authenticate success, global_id 15534 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe881aae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe881aae640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 1 -- 172.21.15.139:0/1023551667 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c031c00 con 0x7fe87c14d1e0 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe881aae640 4 set_mon_vals no callback set 2024-07-26T23:58:53.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.468+0000 7fe87a7fc640 10 monclient: dump: 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:53.486 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:53.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 10 monclient: shutdown 2024-07-26T23:58:53.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:53.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 1 -- 172.21.15.139:0/1023551667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c14d1e0 msgr2=0x7fe87c14f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 1 --2- 172.21.15.139:0/1023551667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c14d1e0 0x7fe87c14f5d0 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0097a0 tx=0x7fe86c02b8d0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 1 -- 172.21.15.139:0/1023551667 shutdown_connections 2024-07-26T23:58:53.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 1 --2- 172.21.15.139:0/1023551667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c14d1e0 0x7fe87c14f5d0 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.490 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 1 -- 172.21.15.139:0/1023551667 >> 172.21.15.139:0/1023551667 conn(0x7fe87c0b62c0 msgr2=0x7fe87c0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 1 -- 172.21.15.139:0/1023551667 shutdown_connections 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.469+0000 7fe881aae640 1 -- 172.21.15.139:0/1023551667 wait complete. 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.470+0000 7fe881aae640 10 monclient: build_initial_monmap 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.470+0000 7fe881aae640 10 monclient: monmap: 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:53.491 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.470+0000 7fe881aae640 1 librados: starting msgr at 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.470+0000 7fe881aae640 1 librados: starting objecter 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.470+0000 7fe881aae640 1 Processor -- start 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 -- start start 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 librados: setting wanted keys 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 librados: calling monclient init 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient: init 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:58:53.492 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe87c0b41c0 0x7fe87c13f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): picked mon.noname-c con 0x7fe87c0b41c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c1470c0 0x7fe87c146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): picked mon.noname-a con 0x7fe87c1470c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe87c14d1e0 0x7fe87c1dbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): picked mon.noname-b con 0x7fe87c14d1e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c141410 con 0x7fe87c1470c0 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c074850 con 0x7fe87c14d1e0 2024-07-26T23:58:53.493 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fe87c141060 con 0x7fe87c0b41c0 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.471+0000 7fe881aae640 10 monclient(hunting): authenticate will time out at 2766.507624s 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c1470c0 0x7fe87c146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87b7fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe87c0b41c0 0x7fe87c13f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe87c14d1e0 0x7fe87c1dbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c1470c0 0x7fe87c146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36280/0 (socket says 172.21.15.139:36280) 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87affd640 1 -- 172.21.15.139:0/1273815214 learned_addr learned my addr 172.21.15.139:0/1273815214 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87affd640 10 monclient(hunting): get_auth_request con 0x7fe87c1470c0 auth_method 0 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87affd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:53.494 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87affd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87b7fe640 10 monclient(hunting): get_auth_request con 0x7fe87c0b41c0 auth_method 0 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87bfff640 10 monclient(hunting): get_auth_request con 0x7fe87c14d1e0 auth_method 0 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.472+0000 7fe87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:53.495 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:53.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:53.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:53.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:53.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:53.496 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.473+0000 7fe87b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:53.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 10 monclient(hunting): handle_auth_done global_id 20269 payload 824 2024-07-26T23:58:53.497 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 1 -- 172.21.15.139:0/1273815214 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe87c0b41c0 msgr2=0x7fe87c13f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe87c0b41c0 0x7fe87c13f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 1 -- 172.21.15.139:0/1273815214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c1470c0 msgr2=0x7fe87c146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c1470c0 0x7fe87c146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 1 monclient: found mon.noname-b 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.475+0000 7fe87bfff640 1 -- 172.21.15.139:0/1273815214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c1416d0 con 0x7fe87c14d1e0 2024-07-26T23:58:53.498 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.477+0000 7fe87bfff640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe87c14d1e0 0x7fe87c1dbfc0 secure :-1 s=READY pgs=4929 cs=0 l=1 rev1=1 crypto rx=0x7fe86c033000 tx=0x7fe86c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.477+0000 7fe873fff640 10 client.?.objecter ms_handle_connect 0x7fe87c14d1e0 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.478+0000 7fe873fff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 1 -- 172.21.15.139:0/1273815214 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c040070 con 0x7fe87c14d1e0 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: dump: 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:53.499 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: _finish_auth 0 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: _check_auth_tickets 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 1 -- 172.21.15.139:0/1273815214 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c03a910 con 0x7fe87c14d1e0 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 1 -- 172.21.15.139:0/1273815214 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c035040 con 0x7fe87c14d1e0 2024-07-26T23:58:53.500 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 5 monclient: authenticate success, global_id 20269 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe873fff640 10 monclient: dump: 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:53.501 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:53.502 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:53.502 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:53.502 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:53.502 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:53.502 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 10 monclient: _renew_subs 2024-07-26T23:58:53.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe879ffb640 4 set_mon_vals no callback set 2024-07-26T23:58:53.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:53.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c1419b0 con 0x7fe87c14d1e0 2024-07-26T23:58:53.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 10 client.20269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:53.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 10 monclient: _renew_subs 2024-07-26T23:58:53.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:53.503 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c1420b0 con 0x7fe87c14d1e0 2024-07-26T23:58:53.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.481+0000 7fe881aae640 1 librados: init done 2024-07-26T23:58:53.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.482+0000 7fe881aae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:53.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.482+0000 7fe881aae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:53.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.482+0000 7fe881aae640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:53.504 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.482+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c0b45e0 con 0x7fe87c14d1e0 2024-07-26T23:58:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.725+0000 7fe873fff640 1 -- 172.21.15.139:0/1273815214 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fe86c038020 con 0x7fe87c14d1e0 2024-07-26T23:58:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.725+0000 7fe873fff640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe85c0789a0 0x7fe85c07ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.725+0000 7fe87b7fe640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe85c0789a0 0x7fe85c07ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.726+0000 7fe87b7fe640 10 monclient: get_auth_request con 0x7fe85c0789a0 auth_method 0 2024-07-26T23:58:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.726+0000 7fe87b7fe640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe85c0789a0 0x7fe85c07ae60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe8740012e0 tx=0x7fe874009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.726+0000 7fe873fff640 10 client.20269.objecter ms_handle_connect 0x7fe85c0789a0 2024-07-26T23:58:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.728+0000 7fe873fff640 1 -- 172.21.15.139:0/1273815214 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(969..969 src has 501..969) ==== 6365+0+0 (secure 0 0 0) 0x7fe86c045d50 con 0x7fe87c14d1e0 2024-07-26T23:58:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.728+0000 7fe873fff640 10 client.20269.objecter ms_dispatch 0x7fe87c000b90 osd_map(969..969 src has 501..969) 2024-07-26T23:58:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.728+0000 7fe873fff640 3 client.20269.objecter handle_osd_map got epochs [969,969] > 0 2024-07-26T23:58:53.744 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.728+0000 7fe873fff640 3 client.20269.objecter handle_osd_map decoding full epoch 969 2024-07-26T23:58:53.744 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.728+0000 7fe873fff640 20 client.20269.objecter dump_active .. 0 homeless 2024-07-26T23:58:53.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.739+0000 7fe873fff640 1 -- 172.21.15.139:0/1273815214 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fe86c082ef0 con 0x7fe87c14d1e0 2024-07-26T23:58:53.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.739+0000 7fe873fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:53.754 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.739+0000 7fe873fff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:53.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.928+0000 7fe881aae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:58:53.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.928+0000 7fe881aae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:58:53.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.928+0000 7fe881aae640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:53.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.928+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe87c141b80 con 0x7fe87c14d1e0 2024-07-26T23:58:53.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.955+0000 7fe873fff640 1 -- 172.21.15.139:0/1273815214 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v969) ==== 74+0+17843 (secure 0 0 0) 0x7fe86c087440 con 0x7fe87c14d1e0 2024-07-26T23:58:53.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.955+0000 7fe873fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-07-26T23:58:53.971 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.955+0000 7fe873fff640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:58:53.971 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:58:53.972 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":969,"fsid":"3b7b1763-526e-44bf-8f07-302d3301e114","created":"2024-07-26T23:34:13.999143+0000","modified":"2024-07-26T23:58:50.488248+0000","last_up_change":"2024-07-26T23:35:14.011640+0000","last_in_change":"2024-07-26T23:34:19.494450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":106,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T23:35:00.571999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-07-26T23:35:15.932143+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-07-26T23:36:22.338751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"26f37aa0-18a9-4f0a-9083-5cfb9a309297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6809","nonce":2123669081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6811","nonce":2123669081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6815","nonce":2123669081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2123669081},{"type":"v1","addr":"172.21.15.139:6813","nonce":2123669081}]},"public_addr":"172.21.15.139:6809/2123669081","cluster_addr":"172.21.15.139:6811/2123669081","heartbeat_back_addr":"172.21.15.139:6815/2123669081","heartbeat_front_addr":"172.21.15.139:6813/2123669081","state":["exists","up"]},{"osd":1,"uuid":"2f5fd361-2a42-47ec-b4ad-b2636b9db59c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6801","nonce":3107328002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6803","nonce":3107328002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6807","nonce":3107328002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":3107328002},{"type":"v1","addr":"172.21.15.139:6805","nonce":3107328002}]},"public_addr":"172.21.15.139:6801/3107328002","cluster_addr":"172.21.15.139:6803/3107328002","heartbeat_back_addr":"172.21.15.139:6807/3107328002","heartbeat_front_addr":"172.21.15.139:6805/3107328002","state":["exists","up"]},{"osd":2,"uuid":"ea342704-f04a-4db8-8519-ed7802ae361f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6825","nonce":1043169956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6827","nonce":1043169956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6831","nonce":1043169956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1043169956},{"type":"v1","addr":"172.21.15.139:6829","nonce":1043169956}]},"public_addr":"172.21.15.139:6825/1043169956","cluster_addr":"172.21.15.139:6827/1043169956","heartbeat_back_addr":"172.21.15.139:6831/1043169956","heartbeat_front_addr":"172.21.15.139:6829/1043169956","state":["exists","up"]},{"osd":3,"uuid":"6f9fd3b7-d9de-4283-aa26-31a80761ebc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6817","nonce":4033735752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6819","nonce":4033735752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6823","nonce":4033735752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4033735752},{"type":"v1","addr":"172.21.15.139:6821","nonce":4033735752}]},"public_addr":"172.21.15.139:6817/4033735752","cluster_addr":"172.21.15.139:6819/4033735752","heartbeat_back_addr":"172.21.15.139:6823/4033735752","heartbeat_front_addr":"172.21.15.139:6821/4033735752","state":["exists","up"]},{"osd":4,"uuid":"5a61bbd3-3522-461c-bfd5-ac5d0d6103ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6827","nonce":4112782318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6829","nonce":4112782318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6833","nonce":4112782318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":4112782318},{"type":"v1","addr":"172.21.15.177:6831","nonce":4112782318}]},"public_addr":"172.21.15.177:6827/4112782318","cluster_addr":"172.21.15.177:6829/4112782318","heartbeat_back_addr":"172.21.15.177:6833/4112782318","heartbeat_front_addr":"172.21.15.177:6831/4112782318","state":["exists","up"]},{"osd":5,"uuid":"add8e38e-b0e0-4fa4-b314-d2e3772398dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6803","nonce":228413488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6805","nonce":228413488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6809","nonce":228413488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":228413488},{"type":"v1","addr":"172.21.15.177:6807","nonce":228413488}]},"public_addr":"172.21.15.177:6803/228413488","cluster_addr":"172.21.15.177:6805/228413488","heartbeat_back_addr":"172.21.15.177:6809/228413488","heartbeat_front_addr":"172.21.15.177:6807/228413488","state":["exists","up"]},{"osd":6,"uuid":"222ad536-af41-44d8-b9ba-464d0f2c6bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6811","nonce":1761005298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6813","nonce":1761005298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6817","nonce":1761005298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1761005298},{"type":"v1","addr":"172.21.15.177:6815","nonce":1761005298}]},"public_addr":"172.21.15.177:6811/1761005298","cluster_addr":"172.21.15.177:6813/1761005298","heartbeat_back_addr":"172.21.15.177:6817/1761005298","heartbeat_front_addr":"172.21.15.177:6815/1761005298","state":["exists","up"]},{"osd":7,"uuid":"5cabe1b4-6f95-4145-8a6f-c92e149c2e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":963,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6819","nonce":3299104561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6821","nonce":3299104561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6825","nonce":3299104561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3299104561},{"type":"v1","addr":"172.21.15.177:6823","nonce":3299104561}]},"public_addr":"172.21.15.177:6819/3299104561","cluster_addr":"172.21.15.177:6821/3299104561","heartbeat_back_addr":"172.21.15.177:6825/3299104561","heartbeat_front_addr":"172.21.15.177:6823/3299104561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:35:00.123097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.190400+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:35:00.557599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:59.842601+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:58.007938+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.331590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.423536+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T23:34:57.831360+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":3,"length":2}]},{"pool":13,"snaps":[{"begin":5,"length":2}]},{"pool":24,"snaps":[{"begin":1,"length":1}]},{"pool":58,"snaps":[{"begin":1,"length":1}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":104,"snaps":[{"begin":7,"length":2}]},{"pool":117,"snaps":[{"begin":1,"length":2}]},{"pool":132,"snaps":[{"begin":3,"length":1}]},{"pool":142,"snaps":[{"begin":2,"length":2}]},{"pool":143,"snaps":[{"begin":2,"length":3}]},{"pool":152,"snaps":[{"begin":2,"length":3}]},{"pool":164,"snaps":[{"begin":2,"length":2}]},{"pool":166,"snaps":[{"begin":4,"length":2}]},{"pool":185,"snaps":[{"begin":2,"length":1}]},{"pool":189,"snaps":[{"begin":2,"length":1}]},{"pool":191,"snaps":[{"begin":2,"length":3}]},{"pool":203,"snaps":[{"begin":4,"length":4}]},{"pool":206,"snaps":[{"begin":3,"length":3}]},{"pool":258,"snaps":[{"begin":7,"length":1}]},{"pool":268,"snaps":[{"begin":8,"length":1}]},{"pool":272,"snaps":[{"begin":10,"length":1}]},{"pool":275,"snaps":[{"begin":11,"length":2}]},{"pool":278,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 10 librados: watch_flush enter 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 10 librados: watch_flush exit 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 20 client.20269.objecter shutdown clearing up homeless session... 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 10 client.20269.objecter successfully canceled tick 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe85c0789a0 msgr2=0x7fe85c07ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe85c0789a0 0x7fe85c07ae60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe8740012e0 tx=0x7fe874009040 comp rx=0 tx=0).stop 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 10 monclient: shutdown 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:53.974 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe87c14d1e0 msgr2=0x7fe87c1dbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe87c14d1e0 0x7fe87c1dbfc0 secure :-1 s=READY pgs=4929 cs=0 l=1 rev1=1 crypto rx=0x7fe86c033000 tx=0x7fe86c0046c0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 shutdown_connections 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe85c0789a0 0x7fe85c07ae60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe87c14d1e0 0x7fe87c1dbfc0 unknown :-1 s=CLOSED pgs=4929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe87c1470c0 0x7fe87c146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 --2- 172.21.15.139:0/1273815214 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe87c0b41c0 0x7fe87c13f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 >> 172.21.15.139:0/1273815214 conn(0x7fe87c0b62c0 msgr2=0x7fe87c0b66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 shutdown_connections 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 -- 172.21.15.139:0/1273815214 wait complete. 2024-07-26T23:58:53.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:53.958+0000 7fe881aae640 1 librados: shutdown 2024-07-26T23:58:54.985 INFO:tasks.ceph:Scrubbing osd.0 2024-07-26T23:58:54.986 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:58:55.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.075+0000 7f8e98ea0640 10 monclient: get_monmap_and_config 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.075+0000 7f8e98ea0640 10 monclient: build_initial_monmap 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.075+0000 7f8e98ea0640 10 monclient: monmap: 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:55.076263+0000 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:55.076263+0000 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.091 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.075+0000 7f8e98ea0640 1 Processor -- start 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 1 -- start start 2024-07-26T23:58:55.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient: init 2024-07-26T23:58:55.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:55.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-07-26T23:58:55.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8e94068740 0x7f8e9413d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient(hunting): picked mon.noname-a con 0x7f8e94068740 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:55.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e9406faf0 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:55.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8e94144090 0x7f8e9413d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient(hunting): picked mon.noname-c con 0x7f8e94144090 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:55.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e940b02c0 con 0x7f8e9406faf0 2024-07-26T23:58:55.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f8e94068b20 con 0x7f8e94068740 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9406d880 con 0x7f8e94144090 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.076+0000 7f8e98ea0640 10 monclient(hunting): authenticate will time out at 2768.112670s 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e9259c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8e94068740 0x7f8e9413d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e92d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8e94144090 0x7f8e9413d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e9259c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8e94068740 0x7f8e9413d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36286/0 (socket says 172.21.15.139:36286) 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e9259c640 1 -- 172.21.15.139:0/2172252869 learned_addr learned my addr 172.21.15.139:0/2172252869 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e9259c640 10 monclient(hunting): get_auth_request con 0x7f8e94068740 auth_method 0 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e9259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.077+0000 7f8e9259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.078+0000 7f8e92d9d640 10 monclient(hunting): get_auth_request con 0x7f8e94144090 auth_method 0 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.078+0000 7f8e92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.078+0000 7f8e92d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.078+0000 7f8e92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.078+0000 7f8e9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.078+0000 7f8e9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.078+0000 7f8e9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 10 monclient(hunting): handle_auth_done global_id 15543 payload 274 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 1 -- 172.21.15.139:0/2172252869 >> v2:172.21.15.177:3300/0 conn(0x7f8e94144090 msgr2=0x7f8e9413d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:55.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 1 --2- 172.21.15.139:0/2172252869 >> v2:172.21.15.177:3300/0 conn(0x7f8e94144090 0x7f8e9413d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 1 -- 172.21.15.139:0/2172252869 >> v1:172.21.15.177:6789/0 conn(0x7f8e9406faf0 legacy=0x7f8e94143980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 10 monclient: _finish_hunting 0 2024-07-26T23:58:55.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 1 monclient: found mon.noname-a 2024-07-26T23:58:55.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:55.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:55.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.079+0000 7f8e9259c640 1 -- 172.21.15.139:0/2172252869 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e941444e0 con 0x7f8e94068740 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.081+0000 7f8e9259c640 1 --2- 172.21.15.139:0/2172252869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 0x7f8e9413d350 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f8e9406ff40 tx=0x7f8e8402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87a58474e8d9a1f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 1 -- 172.21.15.139:0/2172252869 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8402e070 con 0x7f8e94068740 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: dump: 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.101 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: _finish_auth 0 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 1 -- 172.21.15.139:0/2172252869 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e84037070 con 0x7f8e94068740 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 1 -- 172.21.15.139:0/2172252869 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e84031a00 con 0x7f8e94068740 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 5 monclient: authenticate success, global_id 15543 2024-07-26T23:58:55.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 4 set_mon_vals no callback set 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e91d9b640 10 monclient: dump: 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.103 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 10 monclient: shutdown 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 -- 172.21.15.139:0/2172252869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 msgr2=0x7f8e9413d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 --2- 172.21.15.139:0/2172252869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 0x7f8e9413d350 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f8e9406ff40 tx=0x7f8e8402b880 comp rx=0 tx=0).stop 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 -- 172.21.15.139:0/2172252869 shutdown_connections 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 --2- 172.21.15.139:0/2172252869 >> v2:172.21.15.177:3300/0 conn(0x7f8e94144090 0x7f8e9413d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 --2- 172.21.15.139:0/2172252869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 0x7f8e9413d350 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 -- 172.21.15.139:0/2172252869 >> 172.21.15.139:0/2172252869 conn(0x7f8e9406c720 msgr2=0x7f8e9406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 -- 172.21.15.139:0/2172252869 shutdown_connections 2024-07-26T23:58:55.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.085+0000 7f8e98ea0640 1 -- 172.21.15.139:0/2172252869 wait complete. 2024-07-26T23:58:55.107 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient: build_initial_monmap 2024-07-26T23:58:55.107 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient: monmap: 2024-07-26T23:58:55.107 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:55.107 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:55.107 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 librados: starting msgr at 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 librados: starting objecter 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 Processor -- start 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 -- start start 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 librados: setting wanted keys 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 librados: calling monclient init 2024-07-26T23:58:55.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient: init 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 0x7f8e9413e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): picked mon.noname-a con 0x7f8e94068740 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e9406faf0 0x7f8e94071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9406faf0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e94144090 0x7f8e941389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): picked mon.noname-b con 0x7f8e94144090 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9414e290 con 0x7f8e94068740 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941414c0 con 0x7f8e94144090 2024-07-26T23:58:55.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f8e9414e410 con 0x7f8e9406faf0 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.087+0000 7f8e98ea0640 10 monclient(hunting): authenticate will time out at 2768.123988s 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e9259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 0x7f8e9413e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e92d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e94144090 0x7f8e941389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e8bfff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e9406faf0 0x7f8e94071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e9259c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 0x7f8e9413e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36292/0 (socket says 172.21.15.139:36292) 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e9259c640 1 -- 172.21.15.139:0/3609046103 learned_addr learned my addr 172.21.15.139:0/3609046103 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e9259c640 10 monclient(hunting): get_auth_request con 0x7f8e94068740 auth_method 0 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e9259c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e9259c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e92d9d640 10 monclient(hunting): get_auth_request con 0x7f8e94144090 auth_method 0 2024-07-26T23:58:55.110 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e92d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e8bfff640 10 monclient(hunting): get_auth_request con 0x7f8e9406faf0 auth_method 0 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e8bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.088+0000 7f8e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.089+0000 7f8e8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 10 monclient(hunting): handle_auth_done global_id 20275 payload 824 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 1 -- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e9406faf0 msgr2=0x7f8e94071510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e9406faf0 0x7f8e94071510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 1 -- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 msgr2=0x7f8e9413e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e94068740 0x7f8e9413e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 10 monclient: _finish_hunting 0 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 1 monclient: found mon.noname-b 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.092+0000 7f8e92d9d640 1 -- 172.21.15.139:0/3609046103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e94072490 con 0x7f8e94144090 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.094+0000 7f8e92d9d640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e94144090 0x7f8e941389e0 secure :-1 s=READY pgs=4931 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00ebb0 tx=0x7f8e8c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.094+0000 7f8e8affd640 10 client.?.objecter ms_handle_connect 0x7f8e94144090 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.094+0000 7f8e8affd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c018020 con 0x7f8e94144090 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: dump: 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.114 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: _finish_auth 0 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: _check_auth_tickets 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e8c00cd40 con 0x7f8e94144090 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 5 monclient: authenticate success, global_id 20275 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 10 monclient: _renew_subs 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e94072cc0 con 0x7f8e94144090 2024-07-26T23:58:55.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c0146c0 con 0x7f8e94144090 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e8affd640 10 monclient: dump: 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.116 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 10 client.20275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 10 monclient: _renew_subs 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e940729e0 con 0x7f8e94144090 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.098+0000 7f8e98ea0640 1 librados: init done 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.097+0000 7f8e9159a640 4 set_mon_vals no callback set 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.098+0000 7f8e98ea0640 10 client.20275.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.098+0000 7f8e98ea0640 15 client.20275.objecter _session_command_op_assign -1 1 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.098+0000 7f8e98ea0640 20 client.20275.objecter _get_session osd=-1 returning homeless 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.098+0000 7f8e98ea0640 10 client.20275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.098+0000 7f8e98ea0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.098+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e94140910 con 0x7f8e94144090 2024-07-26T23:58:55.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.121+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f8e8c005d10 con 0x7f8e94144090 2024-07-26T23:58:55.137 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.121+0000 7f8e8affd640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e74078a20 0x7f8e7407aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.137 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.122+0000 7f8e9259c640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e74078a20 0x7f8e7407aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.137 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.122+0000 7f8e9259c640 10 monclient: get_auth_request con 0x7f8e74078a20 auth_method 0 2024-07-26T23:58:55.137 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.122+0000 7f8e9259c640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e74078a20 0x7f8e7407aee0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8e8402b8c0 tx=0x7f8e8403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.137 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.122+0000 7f8e8affd640 10 client.20275.objecter ms_handle_connect 0x7f8e74078a20 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.124+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(969..969 src has 501..969) ==== 6365+0+0 (secure 0 0 0) 0x7f8e8c028d80 con 0x7f8e94144090 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.124+0000 7f8e8affd640 10 client.20275.objecter ms_dispatch 0x7f8e94000b90 osd_map(969..969 src has 501..969) 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.124+0000 7f8e8affd640 3 client.20275.objecter handle_osd_map got epochs [969,969] > 0 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.124+0000 7f8e8affd640 3 client.20275.objecter handle_osd_map decoding full epoch 969 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 10 client.20275.objecter checking command 1 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7f8e740807a0 0x7f8e74082c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 20 client.20275.objecter _get_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 20 client.20275.objecter put_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 15 client.20275.objecter _session_command_op_remove -1 1 2024-07-26T23:58:55.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 20 client.20275.objecter _get_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 20 client.20275.objecter get_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 15 client.20275.objecter _session_command_op_assign 0 1 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 20 client.20275.objecter put_session s=0x7f8e740800d0 osd=0 4 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 10 client.20275.objecter _send_command 1 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 --> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e74083310 con 0x7f8e740807a0 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 20 client.20275.objecter dump_active .. 0 homeless 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=969) ==== 24+0+0 (secure 0 0 0) 0x7f8e8c05edf0 con 0x7f8e94144090 2024-07-26T23:58:55.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8affd640 10 monclient: handle_get_version_reply finishing 1 version 969 2024-07-26T23:58:55.142 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.125+0000 7f8e8bfff640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7f8e740807a0 0x7f8e74082c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.142 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.126+0000 7f8e8bfff640 10 monclient: get_auth_request con 0x7f8e740807a0 auth_method 0 2024-07-26T23:58:55.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.128+0000 7f8e8bfff640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7f8e740807a0 0x7f8e74082c40 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.128+0000 7f8e8affd640 10 client.20275.objecter ms_handle_connect 0x7f8e740807a0 2024-07-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.156+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== osd.0 v2:172.21.15.139:6808/2123669081 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e74083310 con 0x7f8e740807a0 2024-07-26T23:58:55.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.156+0000 7f8e8affd640 10 client.20275.objecter ms_dispatch 0x7f8e94000b90 command_reply(tid 1: 0 ) 2024-07-26T23:58:55.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.156+0000 7f8e8affd640 10 client.20275.objecter _finish_command 1 = system:0 2024-07-26T23:58:55.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.156+0000 7f8e8affd640 20 client.20275.objecter put_session s=0x7f8e740800d0 osd=0 4 2024-07-26T23:58:55.172 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.156+0000 7f8e8affd640 15 client.20275.objecter _session_command_op_remove 0 1 2024-07-26T23:58:55.185 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 10 client.20275.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:58:55.185 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 15 client.20275.objecter _session_command_op_assign -1 2 2024-07-26T23:58:55.185 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 20 client.20275.objecter _get_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.185 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 20 client.20275.objecter put_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.185 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 20 client.20275.objecter _get_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 15 client.20275.objecter _session_command_op_remove -1 2 2024-07-26T23:58:55.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 20 client.20275.objecter get_session s=0x7f8e740800d0 osd=0 3 2024-07-26T23:58:55.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 15 client.20275.objecter _session_command_op_assign 0 2 2024-07-26T23:58:55.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 20 client.20275.objecter put_session s=0x7f8e740800d0 osd=0 4 2024-07-26T23:58:55.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 10 client.20275.objecter _send_command 2 2024-07-26T23:58:55.186 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.169+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 --> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8e94143980 con 0x7f8e740807a0 2024-07-26T23:58:55.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.196+0000 7f8e8affd640 1 -- 172.21.15.139:0/3609046103 <== osd.0 v2:172.21.15.139:6808/2123669081 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8e94143980 con 0x7f8e740807a0 2024-07-26T23:58:55.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.196+0000 7f8e8affd640 10 client.20275.objecter ms_dispatch 0x7f8e94000b90 command_reply(tid 2: 0 ) 2024-07-26T23:58:55.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.196+0000 7f8e8affd640 10 client.20275.objecter _finish_command 2 = system:0 2024-07-26T23:58:55.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.196+0000 7f8e8affd640 20 client.20275.objecter put_session s=0x7f8e740800d0 osd=0 4 2024-07-26T23:58:55.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.196+0000 7f8e8affd640 15 client.20275.objecter _session_command_op_remove 0 2 2024-07-26T23:58:55.213 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:58:55.213 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:58:55.213 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:58:55.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 10 librados: watch_flush enter 2024-07-26T23:58:55.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 10 librados: watch_flush exit 2024-07-26T23:58:55.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 10 client.20275.objecter close_session for osd.0 2024-07-26T23:58:55.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7f8e740807a0 msgr2=0x7f8e74082c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.139:6808/2123669081,v1:172.21.15.139:6809/2123669081] conn(0x7f8e740807a0 0x7f8e74082c40 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 20 client.20275.objecter put_session s=0x7f8e740800d0 osd=0 1 2024-07-26T23:58:55.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 20 client.20275.objecter shutdown clearing up homeless session... 2024-07-26T23:58:55.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 10 client.20275.objecter successfully canceled tick 2024-07-26T23:58:55.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e74078a20 msgr2=0x7f8e7407aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e74078a20 0x7f8e7407aee0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8e8402b8c0 tx=0x7f8e8403d040 comp rx=0 tx=0).stop 2024-07-26T23:58:55.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 10 monclient: shutdown 2024-07-26T23:58:55.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:55.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e94144090 msgr2=0x7f8e941389e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 --2- 172.21.15.139:0/3609046103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e94144090 0x7f8e941389e0 secure :-1 s=READY pgs=4931 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00ebb0 tx=0x7f8e8c00c490 comp rx=0 tx=0).stop 2024-07-26T23:58:55.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e92d9d640 1 -- 172.21.15.139:0/3609046103 reap_dead start 2024-07-26T23:58:55.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 shutdown_connections 2024-07-26T23:58:55.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 >> 172.21.15.139:0/3609046103 conn(0x7f8e9406c720 msgr2=0x7f8e9413d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:55.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 shutdown_connections 2024-07-26T23:58:55.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 -- 172.21.15.139:0/3609046103 wait complete. 2024-07-26T23:58:55.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.197+0000 7f8e98ea0640 1 librados: shutdown 2024-07-26T23:58:55.224 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-07-26T23:58:55.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.311+0000 7fcd1b115640 10 monclient: get_monmap_and_config 2024-07-26T23:58:55.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.311+0000 7fcd1b115640 10 monclient: build_initial_monmap 2024-07-26T23:58:55.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.311+0000 7fcd1b115640 10 monclient: monmap: 2024-07-26T23:58:55.326 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:55.326 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.326 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:55.312298+0000 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:55.312298+0000 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:55.327 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 Processor -- start 2024-07-26T23:58:55.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 -- start start 2024-07-26T23:58:55.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient: init 2024-07-26T23:58:55.329 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd1406e720 0x7fcd14139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): picked mon.noname-b con 0x7fcd1406e720 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fcd14143d00 0x7fcd1413a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): picked mon.noname-a con 0x7fcd14143d00 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcd14144cd0 0x7fcd14145140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): picked mon.noname-c con 0x7fcd14144cd0 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1406eb00 con 0x7fcd14143d00 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcd14144710 con 0x7fcd14144cd0 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.330 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1413a650 con 0x7fcd1406e720 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.312+0000 7fcd1b115640 10 monclient(hunting): authenticate will time out at 2768.348787s 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd18eb1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd1406e720 0x7fcd14139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd196b2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcd14144cd0 0x7fcd14145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd18eb1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd1406e720 0x7fcd14139bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48388/0 (socket says 172.21.15.139:48388) 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd196b2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcd14144cd0 0x7fcd14145140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:46864/0 (socket says 172.21.15.139:46864) 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd18eb1640 1 -- 172.21.15.139:0/1406006898 learned_addr learned my addr 172.21.15.139:0/1406006898 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd18eb1640 10 monclient(hunting): get_auth_request con 0x7fcd1406e720 auth_method 0 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd18eb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd18eb1640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd18eb1640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd196b2640 10 monclient(hunting): get_auth_request con 0x7fcd14144cd0 auth_method 0 2024-07-26T23:58:55.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd196b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd196b2640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.313+0000 7fcd196b2640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.314+0000 7fcd196b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.314+0000 7fcd196b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.314+0000 7fcd196b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.314+0000 7fcd18eb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.314+0000 7fcd18eb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.314+0000 7fcd18eb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 10 monclient(hunting): handle_auth_done global_id 20278 payload 274 2024-07-26T23:58:55.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 1 -- 172.21.15.139:0/1406006898 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd1406e720 msgr2=0x7fcd14139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:55.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 1 --2- 172.21.15.139:0/1406006898 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd1406e720 0x7fcd14139bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 1 -- 172.21.15.139:0/1406006898 >> v2:172.21.15.139:3300/0 conn(0x7fcd14143d00 msgr2=0x7fcd1413a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:55.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 1 --2- 172.21.15.139:0/1406006898 >> v2:172.21.15.139:3300/0 conn(0x7fcd14143d00 0x7fcd1413a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 10 monclient: _finish_hunting 0 2024-07-26T23:58:55.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 1 monclient: found mon.noname-c 2024-07-26T23:58:55.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.315+0000 7fcd196b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:55.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.316+0000 7fcd196b2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.316+0000 7fcd196b2640 1 -- 172.21.15.139:0/1406006898 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd140ac810 con 0x7fcd14144cd0 2024-07-26T23:58:55.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.320+0000 7fcd196b2640 1 --2- 172.21.15.139:0/1406006898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14144cd0 0x7fcd14145140 secure :-1 s=READY pgs=4932 cs=0 l=1 rev1=1 crypto rx=0x7fcd1406f1f0 tx=0x7fcd0c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49291f9db1120dab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.340 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 1 -- 172.21.15.139:0/1406006898 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c039070 con 0x7fcd14144cd0 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: dump: 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.341 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 1 -- 172.21.15.139:0/1406006898 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c02cbd0 con 0x7fcd14144cd0 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd1b115640 5 monclient: authenticate success, global_id 20278 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd1b115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd1b115640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 1 -- 172.21.15.139:0/1406006898 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c034440 con 0x7fcd14144cd0 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd1b115640 4 set_mon_vals no callback set 2024-07-26T23:58:55.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.323+0000 7fcd137fe640 10 monclient: dump: 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 10 monclient: shutdown 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 -- 172.21.15.139:0/1406006898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14144cd0 msgr2=0x7fcd14145140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1406006898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14144cd0 0x7fcd14145140 secure :-1 s=READY pgs=4932 cs=0 l=1 rev1=1 crypto rx=0x7fcd1406f1f0 tx=0x7fcd0c02be20 comp rx=0 tx=0).stop 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 -- 172.21.15.139:0/1406006898 shutdown_connections 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1406006898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14144cd0 0x7fcd14145140 unknown :-1 s=CLOSED pgs=4932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1406006898 >> v2:172.21.15.139:3300/0 conn(0x7fcd14143d00 0x7fcd1413a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1406006898 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd1406e720 0x7fcd14139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 -- 172.21.15.139:0/1406006898 >> 172.21.15.139:0/1406006898 conn(0x7fcd140b9c10 msgr2=0x7fcd140ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 -- 172.21.15.139:0/1406006898 shutdown_connections 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.324+0000 7fcd1b115640 1 -- 172.21.15.139:0/1406006898 wait complete. 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 10 monclient: build_initial_monmap 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 10 monclient: monmap: 2024-07-26T23:58:55.344 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:55.345 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 1 librados: starting msgr at 2024-07-26T23:58:55.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 1 librados: starting objecter 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 1 Processor -- start 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 1 -- start start 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 1 librados: setting wanted keys 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 1 librados: calling monclient init 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.325+0000 7fcd1b115640 10 monclient: init 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcd1406e720 0x7fcd14140720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): picked mon.noname-a con 0x7fcd1406e720 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14143d00 0x7fcd1413de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): picked mon.noname-b con 0x7fcd14143d00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd14144cd0 0x7fcd1413e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): picked mon.noname-c con 0x7fcd14144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fcd14141390 con 0x7fcd1406e720 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd141441f0 con 0x7fcd14143d00 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fcd14145680 con 0x7fcd14144cd0 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.326+0000 7fcd1b115640 10 monclient(hunting): authenticate will time out at 2768.362722s 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.331+0000 7fcd196b2640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd14144cd0 0x7fcd1413e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.332+0000 7fcd196b2640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd14144cd0 0x7fcd1413e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48404/0 (socket says 172.21.15.139:48404) 2024-07-26T23:58:55.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.332+0000 7fcd196b2640 1 -- 172.21.15.139:0/1322154141 learned_addr learned my addr 172.21.15.139:0/1322154141 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:55.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.332+0000 7fcd196b2640 10 monclient(hunting): get_auth_request con 0x7fcd14144cd0 auth_method 0 2024-07-26T23:58:55.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.332+0000 7fcd196b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.332+0000 7fcd196b2640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.332+0000 7fcd196b2640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.334+0000 7fcd196b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.334+0000 7fcd196b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.334+0000 7fcd196b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.386+0000 7fcd13fff640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14143d00 0x7fcd1413de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.387+0000 7fcd13fff640 10 monclient(hunting): get_auth_request con 0x7fcd14143d00 auth_method 0 2024-07-26T23:58:55.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.387+0000 7fcd13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.387+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.387+0000 7fcd13fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.387+0000 7fcd13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.387+0000 7fcd13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.387+0000 7fcd13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 10 monclient(hunting): handle_auth_done global_id 20281 payload 824 2024-07-26T23:58:55.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 1 -- 172.21.15.139:0/1322154141 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd14144cd0 msgr2=0x7fcd1413e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd14144cd0 0x7fcd1413e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 1 -- 172.21.15.139:0/1322154141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcd1406e720 msgr2=0x7fcd14140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcd1406e720 0x7fcd14140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.406 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:55.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 1 monclient: found mon.noname-b 2024-07-26T23:58:55.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:55.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.391+0000 7fcd13fff640 1 -- 172.21.15.139:0/1322154141 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd141ce6f0 con 0x7fcd14143d00 2024-07-26T23:58:55.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.395+0000 7fcd13fff640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14143d00 0x7fcd1413de10 secure :-1 s=READY pgs=4933 cs=0 l=1 rev1=1 crypto rx=0x7fcd0000b3c0 tx=0x7fcd0000b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.395+0000 7fcd117fa640 10 client.?.objecter ms_handle_connect 0x7fcd14143d00 2024-07-26T23:58:55.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.395+0000 7fcd117fa640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:55.414 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 1 -- 172.21.15.139:0/1322154141 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00011830 con 0x7fcd14143d00 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: dump: 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.415 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: _finish_auth 0 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: _check_auth_tickets 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 1 -- 172.21.15.139:0/1322154141 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd00016070 con 0x7fcd14143d00 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 5 monclient: authenticate success, global_id 20281 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 10 monclient: _renew_subs 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd141ce9d0 con 0x7fcd14143d00 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 1 -- 172.21.15.139:0/1322154141 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00011c30 con 0x7fcd14143d00 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd117fa640 10 monclient: dump: 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd12ffd640 4 set_mon_vals no callback set 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 10 client.20281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 10 monclient: _renew_subs 2024-07-26T23:58:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd14138e50 con 0x7fcd14143d00 2024-07-26T23:58:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.399+0000 7fcd1b115640 1 librados: init done 2024-07-26T23:58:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.400+0000 7fcd1b115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.400+0000 7fcd1b115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.400+0000 7fcd1b115640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:55.419 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.400+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd14145140 con 0x7fcd14143d00 2024-07-26T23:58:55.664 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.649+0000 7fcd117fa640 1 -- 172.21.15.139:0/1322154141 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fcd0001b460 con 0x7fcd14143d00 2024-07-26T23:58:55.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.649+0000 7fcd117fa640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fccf80789a0 0x7fccf807ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.650+0000 7fcd18eb1640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fccf80789a0 0x7fccf807ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.650+0000 7fcd18eb1640 10 monclient: get_auth_request con 0x7fccf80789a0 auth_method 0 2024-07-26T23:58:55.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.650+0000 7fcd18eb1640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fccf80789a0 0x7fccf807ae60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcd080094a0 tx=0x7fcd080093f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.665 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.650+0000 7fcd117fa640 10 client.20281.objecter ms_handle_connect 0x7fccf80789a0 2024-07-26T23:58:55.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.655+0000 7fcd117fa640 1 -- 172.21.15.139:0/1322154141 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7fcd0005dc80 con 0x7fcd14143d00 2024-07-26T23:58:55.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.655+0000 7fcd117fa640 10 client.20281.objecter ms_dispatch 0x7fcd14000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:55.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.655+0000 7fcd117fa640 3 client.20281.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:55.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.655+0000 7fcd117fa640 3 client.20281.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:55.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.655+0000 7fcd117fa640 20 client.20281.objecter dump_active .. 0 homeless 2024-07-26T23:58:55.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.666+0000 7fcd117fa640 1 -- 172.21.15.139:0/1322154141 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fcd000620f0 con 0x7fcd14143d00 2024-07-26T23:58:55.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.666+0000 7fcd117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:55.680 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.666+0000 7fcd117fa640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:55.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.851+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fcd14143930 con 0x7fccf80789a0 2024-07-26T23:58:55.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.852+0000 7fcd117fa640 1 -- 172.21.15.139:0/1322154141 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcd14143930 con 0x7fccf80789a0 2024-07-26T23:58:55.867 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 0 to deep-scrub 2024-07-26T23:58:55.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 10 librados: watch_flush enter 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 10 librados: watch_flush exit 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 20 client.20281.objecter shutdown clearing up homeless session... 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 10 client.20281.objecter successfully canceled tick 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fccf80789a0 msgr2=0x7fccf807ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fccf80789a0 0x7fccf807ae60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcd080094a0 tx=0x7fcd080093f0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 10 monclient: shutdown 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14143d00 msgr2=0x7fcd1413de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.854+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14143d00 0x7fcd1413de10 secure :-1 s=READY pgs=4933 cs=0 l=1 rev1=1 crypto rx=0x7fcd0000b3c0 tx=0x7fcd0000b880 comp rx=0 tx=0).stop 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 shutdown_connections 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fccf80789a0 0x7fccf807ae60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fcd14144cd0 0x7fcd1413e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd14143d00 0x7fcd1413de10 unknown :-1 s=CLOSED pgs=4933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 --2- 172.21.15.139:0/1322154141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcd1406e720 0x7fcd14140720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 >> 172.21.15.139:0/1322154141 conn(0x7fcd140b9c10 msgr2=0x7fcd140b9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:55.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 shutdown_connections 2024-07-26T23:58:55.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 -- 172.21.15.139:0/1322154141 wait complete. 2024-07-26T23:58:55.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.855+0000 7fcd1b115640 1 librados: shutdown 2024-07-26T23:58:55.881 INFO:tasks.ceph:Scrubbing osd.1 2024-07-26T23:58:55.881 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:58:55.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.973+0000 7f5969677640 10 monclient: get_monmap_and_config 2024-07-26T23:58:55.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.973+0000 7f5969677640 10 monclient: build_initial_monmap 2024-07-26T23:58:55.988 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.973+0000 7f5969677640 10 monclient: monmap: 2024-07-26T23:58:55.988 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:55.988 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.988 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:55.974430+0000 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:55.974430+0000 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:55.989 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.974+0000 7f5969677640 1 Processor -- start 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.974+0000 7f5969677640 1 -- start start 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.974+0000 7f5969677640 10 monclient: init 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.974+0000 7f5969677640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.974+0000 7f5969677640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.974+0000 7f5969677640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f59641470c0 0x7f5964140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.974+0000 7f5969677640 10 monclient(hunting): picked mon.noname-a con 0x7f59641470c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5964140e90 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 10 monclient(hunting): picked mon.noname-b con 0x7f59640b48d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:55.991 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5964147ce0 con 0x7f5964140e90 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f59640acaf0 con 0x7f59641470c0 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f59641477c0 con 0x7f59640b48d0 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f5969677640 10 monclient(hunting): authenticate will time out at 2769.011064s 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f59627fc640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f5964140e90 0x7f59640b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:53928/0 (socket says 172.21.15.139:53928) 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f59627fc640 1 -- 172.21.15.139:0/1993072964 learned_addr learned my addr 172.21.15.139:0/1993072964 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.975+0000 7f59637fe640 1 --2- 172.21.15.139:0/1993072964 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.976+0000 7f59637fe640 10 monclient(hunting): get_auth_request con 0x7f59640b48d0 auth_method 0 2024-07-26T23:58:55.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.976+0000 7f59637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:55.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.976+0000 7f59637fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.993 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.976+0000 7f59637fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.976+0000 7f59637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.976+0000 7f59637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.977+0000 7f59637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.978+0000 7f5961ffb640 1 -- 172.21.15.139:0/1993072964 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495819491 0 0) 0x7f5964147ce0 con 0x7f5964140e90 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.978+0000 7f5961ffb640 10 monclient(hunting): _negotiate 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.978+0000 7f5961ffb640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.978+0000 7f5961ffb640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.978+0000 7f5961ffb640 10 monclient(hunting): my global_id is 20284 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.978+0000 7f5961ffb640 1 -- 172.21.15.139:0/1993072964 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f594c003520 con 0x7f5964140e90 2024-07-26T23:58:55.994 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 10 monclient(hunting): handle_auth_done global_id 16829 payload 274 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 1 -- 172.21.15.139:0/1993072964 >> v2:172.21.15.139:3300/0 conn(0x7f59641470c0 msgr2=0x7f5964140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 1 --2- 172.21.15.139:0/1993072964 >> v2:172.21.15.139:3300/0 conn(0x7f59641470c0 0x7f5964140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 1 -- 172.21.15.139:0/1993072964 >> v1:172.21.15.177:6789/0 conn(0x7f5964140e90 legacy=0x7f59640b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 10 monclient: _finish_hunting 0 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 1 monclient: found mon.noname-b 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:58:55.995 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.979+0000 7f59637fe640 1 -- 172.21.15.139:0/1993072964 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59641474a0 con 0x7f59640b48d0 2024-07-26T23:58:55.996 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.981+0000 7f59637fe640 1 --2- 172.21.15.139:0/1993072964 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596414e1e0 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f595c009d90 tx=0x7f595c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28d8c05839154ea2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:55.998 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5961ffb640 1 -- 172.21.15.139:0/1993072964 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c039070 con 0x7f59640b48d0 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5961ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5961ffb640 10 monclient: dump: 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:55.999 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5961ffb640 10 monclient: _finish_auth 0 2024-07-26T23:58:56.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5961ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:58:56.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:56.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5969677640 5 monclient: authenticate success, global_id 16829 2024-07-26T23:58:56.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.983+0000 7f5969677640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:58:56.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5961ffb640 1 -- 172.21.15.139:0/1993072964 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595c02cbc0 con 0x7f59640b48d0 2024-07-26T23:58:56.000 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5961ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5961ffb640 1 -- 172.21.15.139:0/1993072964 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c034440 con 0x7f59640b48d0 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5961ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5961ffb640 10 monclient: dump: 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.001 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5969677640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:56.002 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.985+0000 7f5969677640 4 set_mon_vals no callback set 2024-07-26T23:58:56.005 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 10 monclient: shutdown 2024-07-26T23:58:56.005 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:56.005 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 -- 172.21.15.139:0/1993072964 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 msgr2=0x7f596414e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.005 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 --2- 172.21.15.139:0/1993072964 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596414e1e0 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f595c009d90 tx=0x7f595c02be10 comp rx=0 tx=0).stop 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 -- 172.21.15.139:0/1993072964 shutdown_connections 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 --2- 172.21.15.139:0/1993072964 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596414e1e0 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 --2- 172.21.15.139:0/1993072964 >> v2:172.21.15.139:3300/0 conn(0x7f59641470c0 0x7f5964140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 -- 172.21.15.139:0/1993072964 >> 172.21.15.139:0/1993072964 conn(0x7f59640b62c0 msgr2=0x7f59640b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 -- 172.21.15.139:0/1993072964 shutdown_connections 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.986+0000 7f5969677640 1 -- 172.21.15.139:0/1993072964 wait complete. 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.987+0000 7f5969677640 10 monclient: build_initial_monmap 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.987+0000 7f5969677640 10 monclient: monmap: 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:56.006 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.987+0000 7f5969677640 1 librados: starting msgr at 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.987+0000 7f5969677640 1 librados: starting objecter 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.987+0000 7f5969677640 1 Processor -- start 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 -- start start 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 librados: setting wanted keys 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 librados: calling monclient init 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient: init 2024-07-26T23:58:56.007 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596413e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): picked mon.noname-c con 0x7f59640b48d0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5964140e90 0x7f59641dc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): picked mon.noname-b con 0x7f5964140e90 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59641470c0 0x7f59641dc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): picked mon.noname-a con 0x7f59641470c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f5964141410 con 0x7f59641470c0 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5964074850 con 0x7f5964140e90 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f596414e720 con 0x7f59640b48d0 2024-07-26T23:58:56.008 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.988+0000 7f5969677640 10 monclient(hunting): authenticate will time out at 2769.024598s 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f5962ffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596413e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5964140e90 0x7f59641dc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59637fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59641470c0 0x7f59641dc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f5962ffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596413e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48414/0 (socket says 172.21.15.139:48414) 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f5962ffd640 1 -- 172.21.15.139:0/3409770087 learned_addr learned my addr 172.21.15.139:0/3409770087 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f5962ffd640 10 monclient(hunting): get_auth_request con 0x7f59640b48d0 auth_method 0 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f5962ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f5962ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f5962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 10 monclient(hunting): get_auth_request con 0x7f5964140e90 auth_method 0 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.009 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59637fe640 10 monclient(hunting): get_auth_request con 0x7f59641470c0 auth_method 0 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59637fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59637fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.989+0000 7f59627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.990+0000 7f5962ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.990+0000 7f5962ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.990+0000 7f5962ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.990+0000 7f59637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.990+0000 7f59637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.010 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.990+0000 7f59637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 10 monclient(hunting): handle_auth_done global_id 20287 payload 824 2024-07-26T23:58:56.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 1 -- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 msgr2=0x7f596413e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.011 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f59640b48d0 0x7f596413e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 1 -- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59641470c0 msgr2=0x7f59641dc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59641470c0 0x7f59641dc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 10 monclient: _finish_hunting 0 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 1 monclient: found mon.noname-b 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.992+0000 7f59627fc640 1 -- 172.21.15.139:0/3409770087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59641dcbd0 con 0x7f5964140e90 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.994+0000 7f59627fc640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5964140e90 0x7f59641dc150 secure :-1 s=READY pgs=4935 cs=0 l=1 rev1=1 crypto rx=0x7f595800c460 tx=0x7f595800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.994+0000 7f594b7fe640 10 client.?.objecter ms_handle_connect 0x7f5964140e90 2024-07-26T23:58:56.012 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.994+0000 7f594b7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:56.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958017070 con 0x7f5964140e90 2024-07-26T23:58:56.013 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: dump: 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.014 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5958005ce0 con 0x7f5964140e90 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5958007270 con 0x7f5964140e90 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 5 monclient: authenticate success, global_id 20287 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 10 monclient: _renew_subs 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596413f8a0 con 0x7f5964140e90 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f594b7fe640 10 monclient: dump: 2024-07-26T23:58:56.015 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f59617fa640 4 set_mon_vals no callback set 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 10 client.20287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 10 monclient: _renew_subs 2024-07-26T23:58:56.016 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5964143b90 con 0x7f5964140e90 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.998+0000 7f5969677640 1 librados: init done 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.999+0000 7f5969677640 10 client.20287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.999+0000 7f5969677640 15 client.20287.objecter _session_command_op_assign -1 1 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.999+0000 7f5969677640 20 client.20287.objecter _get_session osd=-1 returning homeless 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.999+0000 7f5969677640 10 client.20287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.999+0000 7f5969677640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.017 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:55.999+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5964143dc0 con 0x7f5964140e90 2024-07-26T23:58:56.033 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.018+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f595801a070 con 0x7f5964140e90 2024-07-26T23:58:56.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.019+0000 7f594b7fe640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f59500789a0 0x7f595007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.019+0000 7f5962ffd640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f59500789a0 0x7f595007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.019+0000 7f5962ffd640 10 monclient: get_auth_request con 0x7f59500789a0 auth_method 0 2024-07-26T23:58:56.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.019+0000 7f594b7fe640 10 client.20287.objecter ms_handle_connect 0x7f59500789a0 2024-07-26T23:58:56.034 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.019+0000 7f5962ffd640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f59500789a0 0x7f595007ae60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f59540099a0 tx=0x7f5954006f90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f5958096590 con 0x7f5964140e90 2024-07-26T23:58:56.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 10 client.20287.objecter ms_dispatch 0x7f5964000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:56.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 3 client.20287.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:56.038 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 3 client.20287.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 10 client.20287.objecter checking command 1 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f5950080720 0x7f5950082bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 20 client.20287.objecter _get_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 20 client.20287.objecter put_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 15 client.20287.objecter _session_command_op_remove -1 1 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 20 client.20287.objecter _get_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 20 client.20287.objecter get_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 15 client.20287.objecter _session_command_op_assign 1 1 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 20 client.20287.objecter put_session s=0x7f5950080050 osd=1 4 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 10 client.20287.objecter _send_command 1 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5950083290 con 0x7f5950080720 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 20 client.20287.objecter dump_active .. 0 homeless 2024-07-26T23:58:56.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=970) ==== 24+0+0 (secure 0 0 0) 0x7f595805cb60 con 0x7f5964140e90 2024-07-26T23:58:56.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f594b7fe640 10 monclient: handle_get_version_reply finishing 1 version 970 2024-07-26T23:58:56.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.023+0000 7f59637fe640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f5950080720 0x7f5950082bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.024+0000 7f59637fe640 10 monclient: get_auth_request con 0x7f5950080720 auth_method 0 2024-07-26T23:58:56.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.027+0000 7f594b7fe640 10 client.20287.objecter ms_handle_connect 0x7f5950080720 2024-07-26T23:58:56.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.027+0000 7f59637fe640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f5950080720 0x7f5950082bc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.051+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== osd.1 v2:172.21.15.139:6800/3107328002 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5950083290 con 0x7f5950080720 2024-07-26T23:58:56.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.051+0000 7f594b7fe640 10 client.20287.objecter ms_dispatch 0x7f5964000b90 command_reply(tid 1: 0 ) 2024-07-26T23:58:56.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.051+0000 7f594b7fe640 10 client.20287.objecter _finish_command 1 = system:0 2024-07-26T23:58:56.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.051+0000 7f594b7fe640 20 client.20287.objecter put_session s=0x7f5950080050 osd=1 4 2024-07-26T23:58:56.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.051+0000 7f594b7fe640 15 client.20287.objecter _session_command_op_remove 1 1 2024-07-26T23:58:56.081 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 10 client.20287.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:58:56.081 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 15 client.20287.objecter _session_command_op_assign -1 2 2024-07-26T23:58:56.081 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 20 client.20287.objecter _get_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.081 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 20 client.20287.objecter put_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 20 client.20287.objecter _get_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 15 client.20287.objecter _session_command_op_remove -1 2 2024-07-26T23:58:56.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 20 client.20287.objecter get_session s=0x7f5950080050 osd=1 3 2024-07-26T23:58:56.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 15 client.20287.objecter _session_command_op_assign 1 2 2024-07-26T23:58:56.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 20 client.20287.objecter put_session s=0x7f5950080050 osd=1 4 2024-07-26T23:58:56.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 10 client.20287.objecter _send_command 2 2024-07-26T23:58:56.082 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.066+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 --> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f596413e500 con 0x7f5950080720 2024-07-26T23:58:56.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f594b7fe640 1 -- 172.21.15.139:0/3409770087 <== osd.1 v2:172.21.15.139:6800/3107328002 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f595c03a040 con 0x7f5950080720 2024-07-26T23:58:56.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f594b7fe640 10 client.20287.objecter ms_dispatch 0x7f5964000b90 command_reply(tid 2: 0 ) 2024-07-26T23:58:56.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f594b7fe640 10 client.20287.objecter _finish_command 2 = system:0 2024-07-26T23:58:56.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f594b7fe640 20 client.20287.objecter put_session s=0x7f5950080050 osd=1 4 2024-07-26T23:58:56.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f594b7fe640 15 client.20287.objecter _session_command_op_remove 1 2 2024-07-26T23:58:56.099 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:58:56.099 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:58:56.099 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 10 librados: watch_flush enter 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 10 librados: watch_flush exit 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 10 client.20287.objecter close_session for osd.1 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f5950080720 msgr2=0x7f5950082bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.139:6800/3107328002,v1:172.21.15.139:6801/3107328002] conn(0x7f5950080720 0x7f5950082bc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 20 client.20287.objecter put_session s=0x7f5950080050 osd=1 1 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 20 client.20287.objecter shutdown clearing up homeless session... 2024-07-26T23:58:56.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 10 client.20287.objecter successfully canceled tick 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f59500789a0 msgr2=0x7f595007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f59500789a0 0x7f595007ae60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f59540099a0 tx=0x7f5954006f90 comp rx=0 tx=0).stop 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 10 monclient: shutdown 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5964140e90 msgr2=0x7f59641dc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 --2- 172.21.15.139:0/3409770087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5964140e90 0x7f59641dc150 secure :-1 s=READY pgs=4935 cs=0 l=1 rev1=1 crypto rx=0x7f595800c460 tx=0x7f595800c920 comp rx=0 tx=0).stop 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f59637fe640 1 -- 172.21.15.139:0/3409770087 reap_dead start 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 shutdown_connections 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 >> 172.21.15.139:0/3409770087 conn(0x7f59640b62c0 msgr2=0x7f5964068c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 shutdown_connections 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.083+0000 7f5969677640 1 -- 172.21.15.139:0/3409770087 wait complete. 2024-07-26T23:58:56.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.084+0000 7f5969677640 1 librados: shutdown 2024-07-26T23:58:56.109 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-07-26T23:58:56.209 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.194+0000 7ff920aa8640 10 monclient: get_monmap_and_config 2024-07-26T23:58:56.209 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.194+0000 7ff920aa8640 10 monclient: build_initial_monmap 2024-07-26T23:58:56.209 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.194+0000 7ff920aa8640 10 monclient: monmap: 2024-07-26T23:58:56.209 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:56.195517+0000 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:56.195517+0000 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:56.210 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 1 Processor -- start 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 1 -- start start 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient: init 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7ff91c0687a0 0x7ff91c13d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient(hunting): picked mon.noname-a con 0x7ff91c0687a0 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff91c06fa60 0x7ff91c13d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient(hunting): picked mon.noname-c con 0x7ff91c06fa60 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff91c143910 addr v1:172.21.15.139:6789/0 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91c0af960 con 0x7ff91c143910 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7ff91c13dd40 con 0x7ff91c0687a0 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff91c0b0230 con 0x7ff91c06fa60 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.195+0000 7ff920aa8640 10 monclient(hunting): authenticate will time out at 2769.231721s 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff91a59c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7ff91c0687a0 0x7ff91c13d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff911d9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff91c06fa60 0x7ff91c13d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff911d9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff91c06fa60 0x7ff91c13d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:46894/0 (socket says 172.21.15.139:46894) 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff911d9b640 1 -- 172.21.15.139:0/1343908025 learned_addr learned my addr 172.21.15.139:0/1343908025 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff911d9b640 10 monclient(hunting): get_auth_request con 0x7ff91c06fa60 auth_method 0 2024-07-26T23:58:56.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff911d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff911d9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff911d9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.196+0000 7ff91a59c640 10 monclient(hunting): get_auth_request con 0x7ff91c0687a0 auth_method 0 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff91a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff91a59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff91a59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff911d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff911d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff911d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff91a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff91a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.197+0000 7ff91a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 10 monclient(hunting): handle_auth_done global_id 20290 payload 274 2024-07-26T23:58:56.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 1 -- 172.21.15.139:0/1343908025 >> v2:172.21.15.139:3300/0 conn(0x7ff91c0687a0 msgr2=0x7ff91c13d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:56.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 1 --2- 172.21.15.139:0/1343908025 >> v2:172.21.15.139:3300/0 conn(0x7ff91c0687a0 0x7ff91c13d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 1 -- 172.21.15.139:0/1343908025 >> v1:172.21.15.139:6789/0 conn(0x7ff91c143910 legacy=0x7ff91c143d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 10 monclient: _finish_hunting 0 2024-07-26T23:58:56.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 1 monclient: found mon.noname-c 2024-07-26T23:58:56.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:56.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.198+0000 7ff911d9b640 1 -- 172.21.15.139:0/1343908025 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91c06d950 con 0x7ff91c06fa60 2024-07-26T23:58:56.217 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.200+0000 7ff911d9b640 1 --2- 172.21.15.139:0/1343908025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c06fa60 0x7ff91c13d800 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 crypto rx=0x7ff91c06b960 tx=0x7ff90402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0f5a056580f5f36 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 1 -- 172.21.15.139:0/1343908025 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904035020 con 0x7ff91c06fa60 2024-07-26T23:58:56.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:58:56.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: dump: 2024-07-26T23:58:56.220 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.220 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.220 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: _finish_auth 0 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 1 -- 172.21.15.139:0/1343908025 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff904031870 con 0x7ff91c06fa60 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 1 -- 172.21.15.139:0/1343908025 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904031c40 con 0x7ff91c06fa60 2024-07-26T23:58:56.221 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff919d9b640 10 monclient: dump: 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.222 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff920aa8640 5 monclient: authenticate success, global_id 20290 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff920aa8640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.204+0000 7ff920aa8640 4 set_mon_vals no callback set 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 10 monclient: shutdown 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 -- 172.21.15.139:0/1343908025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c06fa60 msgr2=0x7ff91c13d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 --2- 172.21.15.139:0/1343908025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c06fa60 0x7ff91c13d800 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 crypto rx=0x7ff91c06b960 tx=0x7ff90402b910 comp rx=0 tx=0).stop 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 -- 172.21.15.139:0/1343908025 shutdown_connections 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 --2- 172.21.15.139:0/1343908025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c06fa60 0x7ff91c13d800 unknown :-1 s=CLOSED pgs=4936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 --2- 172.21.15.139:0/1343908025 >> v2:172.21.15.139:3300/0 conn(0x7ff91c0687a0 0x7ff91c13d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 -- 172.21.15.139:0/1343908025 >> 172.21.15.139:0/1343908025 conn(0x7ff91c06c720 msgr2=0x7ff91c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 -- 172.21.15.139:0/1343908025 shutdown_connections 2024-07-26T23:58:56.223 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 1 -- 172.21.15.139:0/1343908025 wait complete. 2024-07-26T23:58:56.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 10 monclient: build_initial_monmap 2024-07-26T23:58:56.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.205+0000 7ff920aa8640 10 monclient: monmap: 2024-07-26T23:58:56.226 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:56.226 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:56.226 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:56.226 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:56.226 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:56.340 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.340 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:56.340 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:56.340 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:56.340 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.340 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.206+0000 7ff920aa8640 1 librados: starting msgr at 2024-07-26T23:58:56.340 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.206+0000 7ff920aa8640 1 librados: starting objecter 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.206+0000 7ff920aa8640 1 Processor -- start 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.206+0000 7ff920aa8640 1 -- start start 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.206+0000 7ff920aa8640 1 librados: setting wanted keys 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.206+0000 7ff920aa8640 1 librados: calling monclient init 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.206+0000 7ff920aa8640 10 monclient: init 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff91c0687a0 0x7ff91c073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): picked mon.noname-c con 0x7ff91c0687a0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff91c06fa60 0x7ff91c074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): picked mon.noname-a con 0x7ff91c06fa60 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c143910 0x7ff91c139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): picked mon.noname-b con 0x7ff91c143910 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7ff91c14e220 con 0x7ff91c06fa60 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff91c141450 con 0x7ff91c143910 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7ff91c14e3a0 con 0x7ff91c0687a0 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff920aa8640 10 monclient(hunting): authenticate will time out at 2769.243124s 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.207+0000 7ff91a59c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff91c0687a0 0x7ff91c073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff91c0687a0 0x7ff91c073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48422/0 (socket says 172.21.15.139:48422) 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 1 -- 172.21.15.139:0/4169078623 learned_addr learned my addr 172.21.15.139:0/4169078623 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:56.342 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91ad9d640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c143910 0x7ff91c139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 10 monclient(hunting): get_auth_request con 0x7ff91c0687a0 auth_method 0 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91ad9d640 10 monclient(hunting): get_auth_request con 0x7ff91c143910 auth_method 0 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91ad9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.208+0000 7ff91ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.209+0000 7ff91ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.209+0000 7ff91ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.343 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.209+0000 7ff91ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 10 monclient(hunting): handle_auth_done global_id 20293 payload 824 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 1 -- 172.21.15.139:0/4169078623 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff91c0687a0 msgr2=0x7ff91c073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff91c0687a0 0x7ff91c073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 1 -- 172.21.15.139:0/4169078623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff91c06fa60 msgr2=0x7ff91c074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff91c06fa60 0x7ff91c074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 10 monclient: _finish_hunting 0 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 1 monclient: found mon.noname-b 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:56.346 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.211+0000 7ff91ad9d640 1 -- 172.21.15.139:0/4169078623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91c073390 con 0x7ff91c143910 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.213+0000 7ff91ad9d640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c143910 0x7ff91c139160 secure :-1 s=READY pgs=4937 cs=0 l=1 rev1=1 crypto rx=0x7ff91400c760 tx=0x7ff91400cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.213+0000 7ff9137fe640 10 client.?.objecter ms_handle_connect 0x7ff91c143910 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.213+0000 7ff9137fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 1 -- 172.21.15.139:0/4169078623 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914003b70 con 0x7ff91c143910 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: dump: 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.347 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 1 -- 172.21.15.139:0/4169078623 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff914003d10 con 0x7ff91c143910 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 1 -- 172.21.15.139:0/4169078623 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140052f0 con 0x7ff91c143910 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 5 monclient: authenticate success, global_id 20293 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 10 monclient: _renew_subs 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.348 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff91c147b10 con 0x7ff91c143910 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff9137fe640 10 monclient: dump: 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.349 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.350 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff91959a640 4 set_mon_vals no callback set 2024-07-26T23:58:56.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 10 client.20293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:56.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 10 monclient: _renew_subs 2024-07-26T23:58:56.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff91c139880 con 0x7ff91c143910 2024-07-26T23:58:56.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.217+0000 7ff920aa8640 1 librados: init done 2024-07-26T23:58:56.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.218+0000 7ff920aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:56.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.218+0000 7ff920aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:56.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.218+0000 7ff920aa8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.218+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff91c143d60 con 0x7ff91c143910 2024-07-26T23:58:56.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.469+0000 7ff9137fe640 1 -- 172.21.15.139:0/4169078623 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7ff914005490 con 0x7ff91c143910 2024-07-26T23:58:56.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.470+0000 7ff9137fe640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff8fc0789f0 0x7ff8fc07aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.470+0000 7ff91a59c640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff8fc0789f0 0x7ff8fc07aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.470+0000 7ff91a59c640 10 monclient: get_auth_request con 0x7ff8fc0789f0 auth_method 0 2024-07-26T23:58:56.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.470+0000 7ff9137fe640 10 client.20293.objecter ms_handle_connect 0x7ff8fc0789f0 2024-07-26T23:58:56.485 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.470+0000 7ff91a59c640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff8fc0789f0 0x7ff8fc07aeb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009c00 tx=0x7ff90c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.473+0000 7ff9137fe640 1 -- 172.21.15.139:0/4169078623 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7ff914005780 con 0x7ff91c143910 2024-07-26T23:58:56.488 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.473+0000 7ff9137fe640 10 client.20293.objecter ms_dispatch 0x7ff91c000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:56.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.473+0000 7ff9137fe640 3 client.20293.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:56.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.473+0000 7ff9137fe640 3 client.20293.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:56.489 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.473+0000 7ff9137fe640 20 client.20293.objecter dump_active .. 0 homeless 2024-07-26T23:58:56.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.484+0000 7ff9137fe640 1 -- 172.21.15.139:0/4169078623 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7ff9140621b0 con 0x7ff91c143910 2024-07-26T23:58:56.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.484+0000 7ff9137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:56.499 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.484+0000 7ff9137fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:56.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.708+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7ff91c147ce0 con 0x7ff8fc0789f0 2024-07-26T23:58:56.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.709+0000 7ff9137fe640 1 -- 172.21.15.139:0/4169078623 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff91c147ce0 con 0x7ff8fc0789f0 2024-07-26T23:58:56.724 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 1 to deep-scrub 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 10 librados: watch_flush enter 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 10 librados: watch_flush exit 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 20 client.20293.objecter shutdown clearing up homeless session... 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 10 client.20293.objecter successfully canceled tick 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff8fc0789f0 msgr2=0x7ff8fc07aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff8fc0789f0 0x7ff8fc07aeb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009c00 tx=0x7ff90c009190 comp rx=0 tx=0).stop 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 10 monclient: shutdown 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c143910 msgr2=0x7ff91c139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c143910 0x7ff91c139160 secure :-1 s=READY pgs=4937 cs=0 l=1 rev1=1 crypto rx=0x7ff91400c760 tx=0x7ff91400cc20 comp rx=0 tx=0).stop 2024-07-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 shutdown_connections 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7ff8fc0789f0 0x7ff8fc07aeb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff91c143910 0x7ff91c139160 unknown :-1 s=CLOSED pgs=4937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff91c06fa60 0x7ff91c074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 --2- 172.21.15.139:0/4169078623 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7ff91c0687a0 0x7ff91c073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 >> 172.21.15.139:0/4169078623 conn(0x7ff91c06c720 msgr2=0x7ff91c13b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 shutdown_connections 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 -- 172.21.15.139:0/4169078623 wait complete. 2024-07-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.711+0000 7ff920aa8640 1 librados: shutdown 2024-07-26T23:58:56.739 INFO:tasks.ceph:Scrubbing osd.2 2024-07-26T23:58:56.739 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:58:56.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.827+0000 7f6cfd6dd640 10 monclient: get_monmap_and_config 2024-07-26T23:58:56.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.827+0000 7f6cfd6dd640 10 monclient: build_initial_monmap 2024-07-26T23:58:56.842 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.827+0000 7f6cfd6dd640 10 monclient: monmap: 2024-07-26T23:58:56.842 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:56.842 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:56.828367+0000 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:56.828367+0000 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:56.843 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:56.844 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:56.844 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.827+0000 7f6cfd6dd640 1 Processor -- start 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 1 -- start start 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient: init 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cf81470c0 addr v1:172.21.15.139:6789/0 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6cf80b41c0 0x7f6cf80b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.845 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f6cf80b41c0 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6cf814c1d0 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf8147ce0 con 0x7f6cf81470c0 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf80acaf0 con 0x7f6cf814c1d0 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf81477c0 con 0x7f6cf80b41c0 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.828+0000 7f6cfd6dd640 10 monclient(hunting): authenticate will time out at 2769.864674s 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.829+0000 7f6cf6ffd640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f6cf81470c0 0x7f6cf8140950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:44784/0 (socket says 172.21.15.139:44784) 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.829+0000 7f6cf6ffd640 1 -- 172.21.15.139:0/497465611 learned_addr learned my addr 172.21.15.139:0/497465611 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.829+0000 7f6cf67fc640 1 --2- 172.21.15.139:0/497465611 >> v2:172.21.15.177:3300/0 conn(0x7f6cf80b41c0 0x7f6cf80b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.829+0000 7f6cf67fc640 10 monclient(hunting): get_auth_request con 0x7f6cf80b41c0 auth_method 0 2024-07-26T23:58:56.846 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.829+0000 7f6cf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.829+0000 7f6cf67fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.829+0000 7f6cf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.830+0000 7f6cf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.830+0000 7f6cf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.847 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.830+0000 7f6cf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.831+0000 7f6cf5ffb640 1 -- 172.21.15.139:0/497465611 <== mon.0 v1:172.21.15.139:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4270685588 0 0) 0x7f6cf8147ce0 con 0x7f6cf81470c0 2024-07-26T23:58:56.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.831+0000 7f6cf5ffb640 10 monclient(hunting): _negotiate 2024-07-26T23:58:56.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.831+0000 7f6cf5ffb640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.831+0000 7f6cf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.831+0000 7f6cf5ffb640 10 monclient(hunting): my global_id is 15555 2024-07-26T23:58:56.848 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.831+0000 7f6cf5ffb640 1 -- 172.21.15.139:0/497465611 --> v1:172.21.15.139:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ce4003520 con 0x7f6cf81470c0 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 10 monclient(hunting): handle_auth_done global_id 20296 payload 274 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 1 -- 172.21.15.139:0/497465611 >> v1:172.21.15.177:6789/0 conn(0x7f6cf814c1d0 legacy=0x7f6cf814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 1 -- 172.21.15.139:0/497465611 >> v1:172.21.15.139:6789/0 conn(0x7f6cf81470c0 legacy=0x7f6cf8140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 10 monclient: _finish_hunting 0 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 1 monclient: found mon.noname-c 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.849 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.834+0000 7f6cf67fc640 1 -- 172.21.15.139:0/497465611 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf81474a0 con 0x7f6cf80b41c0 2024-07-26T23:58:56.852 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.837+0000 7f6cf67fc640 1 --2- 172.21.15.139:0/497465611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf80b45e0 secure :-1 s=READY pgs=4939 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=decbe4553ccb363d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 1 -- 172.21.15.139:0/497465611 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0035020 con 0x7f6cf80b41c0 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 10 monclient: dump: 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.856 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 10 monclient: _finish_auth 0 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.840+0000 7f6cf5ffb640 1 -- 172.21.15.139:0/497465611 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0031830 con 0x7f6cf80b41c0 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cf5ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cf5ffb640 1 -- 172.21.15.139:0/497465611 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0031c00 con 0x7f6cf80b41c0 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.857 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cf5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cf5ffb640 10 monclient: dump: 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 5 monclient: authenticate success, global_id 20296 2024-07-26T23:58:56.858 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:56.859 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 4 set_mon_vals no callback set 2024-07-26T23:58:56.861 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 10 monclient: shutdown 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/497465611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 msgr2=0x7f6cf80b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 1 --2- 172.21.15.139:0/497465611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf80b45e0 secure :-1 s=READY pgs=4939 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b8d0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/497465611 shutdown_connections 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 1 --2- 172.21.15.139:0/497465611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf80b45e0 unknown :-1 s=CLOSED pgs=4939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/497465611 >> 172.21.15.139:0/497465611 conn(0x7f6cf80b62c0 msgr2=0x7f6cf80b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/497465611 shutdown_connections 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.841+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/497465611 wait complete. 2024-07-26T23:58:56.862 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.842+0000 7f6cfd6dd640 10 monclient: build_initial_monmap 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.842+0000 7f6cfd6dd640 10 monclient: monmap: 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.842+0000 7f6cfd6dd640 1 librados: starting msgr at 2024-07-26T23:58:56.863 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.842+0000 7f6cfd6dd640 1 librados: starting objecter 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 Processor -- start 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 -- start start 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 librados: setting wanted keys 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 librados: calling monclient init 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient: init 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf8143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.864 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f6cf80b41c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6cf81470c0 0x7f6cf80af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f6cf81470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cf814c1d0 0x7f6cf80af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f6cf814c1d0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8141410 con 0x7f6cf814c1d0 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8074850 con 0x7f6cf80b41c0 2024-07-26T23:58:56.865 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf80b4b20 con 0x7f6cf81470c0 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.843+0000 7f6cfd6dd640 10 monclient(hunting): authenticate will time out at 2769.879643s 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf8143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf67fc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6cf81470c0 0x7f6cf80af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf77fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cf814c1d0 0x7f6cf80af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf8143a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:46922/0 (socket says 172.21.15.139:46922) 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf6ffd640 1 -- 172.21.15.139:0/3178611168 learned_addr learned my addr 172.21.15.139:0/3178611168 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf6ffd640 10 monclient(hunting): get_auth_request con 0x7f6cf80b41c0 auth_method 0 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.866 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf77fe640 10 monclient(hunting): get_auth_request con 0x7f6cf814c1d0 auth_method 0 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf77fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf67fc640 10 monclient(hunting): get_auth_request con 0x7f6cf81470c0 auth_method 0 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf67fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.844+0000 7f6cf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.867 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.868 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.868 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:56.868 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:56.868 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.845+0000 7f6cf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:56.869 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 10 monclient(hunting): handle_auth_done global_id 20302 payload 824 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 1 -- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6cf81470c0 msgr2=0x7f6cf80af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f6cf81470c0 0x7f6cf80af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 1 -- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cf814c1d0 msgr2=0x7f6cf80af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6cf814c1d0 0x7f6cf80af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 1 monclient: found mon.noname-b 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.847+0000 7f6cf6ffd640 1 -- 172.21.15.139:0/3178611168 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf80b0670 con 0x7f6cf80b41c0 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.850+0000 7f6cf6ffd640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf8143a70 secure :-1 s=READY pgs=4940 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800cbe0 tx=0x7f6ce800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.850+0000 7f6cef7fe640 10 client.?.objecter ms_handle_connect 0x7f6cf80b41c0 2024-07-26T23:58:56.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.850+0000 7f6cef7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8017070 con 0x7f6cf80b41c0 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: dump: 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.871 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.872 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:56.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:56.872 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce800ad60 con 0x7f6cf80b41c0 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8012640 con 0x7f6cf80b41c0 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 5 monclient: authenticate success, global_id 20302 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 10 monclient: _renew_subs 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf80b0950 con 0x7f6cf80b41c0 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cef7fe640 10 monclient: dump: 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:56.873 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cf57fa640 4 set_mon_vals no callback set 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 10 client.20302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 10 monclient: _renew_subs 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf80b10c0 con 0x7f6cf80b41c0 2024-07-26T23:58:56.874 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.854+0000 7f6cfd6dd640 1 librados: init done 2024-07-26T23:58:56.875 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.855+0000 7f6cfd6dd640 10 client.20302.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:56.875 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.855+0000 7f6cfd6dd640 15 client.20302.objecter _session_command_op_assign -1 1 2024-07-26T23:58:56.875 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.855+0000 7f6cfd6dd640 20 client.20302.objecter _get_session osd=-1 returning homeless 2024-07-26T23:58:56.875 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.855+0000 7f6cfd6dd640 10 client.20302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:56.875 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.855+0000 7f6cfd6dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:56.875 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.855+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6cf80b45e0 con 0x7f6cf80b41c0 2024-07-26T23:58:56.890 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.875+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f6ce801a070 con 0x7f6cf80b41c0 2024-07-26T23:58:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.876+0000 7f6cef7fe640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6cd8078a20 0x7f6cd807aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.876+0000 7f6cf67fc640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6cd8078a20 0x7f6cd807aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.876+0000 7f6cf67fc640 10 monclient: get_auth_request con 0x7f6cd8078a20 auth_method 0 2024-07-26T23:58:56.892 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.877+0000 7f6cef7fe640 10 client.20302.objecter ms_handle_connect 0x7f6cd8078a20 2024-07-26T23:58:56.892 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.877+0000 7f6cf67fc640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6cd8078a20 0x7f6cd807aee0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0036000 tx=0x7f6ce003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f6ce8007940 con 0x7f6cf80b41c0 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 10 client.20302.objecter ms_dispatch 0x7f6cf8000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 3 client.20302.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 3 client.20302.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 10 client.20302.objecter checking command 1 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f6cd8080800 0x7f6cd8082ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 20 client.20302.objecter _get_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 20 client.20302.objecter put_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 15 client.20302.objecter _session_command_op_remove -1 1 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 20 client.20302.objecter _get_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 20 client.20302.objecter get_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 15 client.20302.objecter _session_command_op_assign 2 1 2024-07-26T23:58:56.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 20 client.20302.objecter put_session s=0x7f6cd8080130 osd=2 4 2024-07-26T23:58:56.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 10 client.20302.objecter _send_command 1 2024-07-26T23:58:56.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 --> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6cd8083370 con 0x7f6cd8080800 2024-07-26T23:58:56.896 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.879+0000 7f6cef7fe640 20 client.20302.objecter dump_active .. 0 homeless 2024-07-26T23:58:56.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.880+0000 7f6cf77fe640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f6cd8080800 0x7f6cd8082ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:56.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.880+0000 7f6cf77fe640 10 monclient: get_auth_request con 0x7f6cd8080800 auth_method 0 2024-07-26T23:58:56.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.881+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=970) ==== 24+0+0 (secure 0 0 0) 0x7f6ce805de60 con 0x7f6cf80b41c0 2024-07-26T23:58:56.897 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.881+0000 7f6cef7fe640 10 monclient: handle_get_version_reply finishing 1 version 970 2024-07-26T23:58:56.898 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.883+0000 7f6cf77fe640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f6cd8080800 0x7f6cd8082ca0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:56.898 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.883+0000 7f6cef7fe640 10 client.20302.objecter ms_handle_connect 0x7f6cd8080800 2024-07-26T23:58:56.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.906+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== osd.2 v2:172.21.15.139:6824/1043169956 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6cd8083370 con 0x7f6cd8080800 2024-07-26T23:58:56.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.906+0000 7f6cef7fe640 10 client.20302.objecter ms_dispatch 0x7f6cf8000b90 command_reply(tid 1: 0 ) 2024-07-26T23:58:56.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.906+0000 7f6cef7fe640 10 client.20302.objecter _finish_command 1 = system:0 2024-07-26T23:58:56.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.906+0000 7f6cef7fe640 20 client.20302.objecter put_session s=0x7f6cd8080130 osd=2 4 2024-07-26T23:58:56.921 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.906+0000 7f6cef7fe640 15 client.20302.objecter _session_command_op_remove 2 1 2024-07-26T23:58:56.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 10 client.20302.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:58:56.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 15 client.20302.objecter _session_command_op_assign -1 2 2024-07-26T23:58:56.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 20 client.20302.objecter _get_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 20 client.20302.objecter put_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 20 client.20302.objecter _get_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 15 client.20302.objecter _session_command_op_remove -1 2 2024-07-26T23:58:56.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 20 client.20302.objecter get_session s=0x7f6cd8080130 osd=2 3 2024-07-26T23:58:56.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 15 client.20302.objecter _session_command_op_assign 2 2 2024-07-26T23:58:56.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 20 client.20302.objecter put_session s=0x7f6cd8080130 osd=2 4 2024-07-26T23:58:56.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 10 client.20302.objecter _send_command 2 2024-07-26T23:58:56.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.919+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 --> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6cf80b8dc0 con 0x7f6cd8080800 2024-07-26T23:58:56.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cef7fe640 1 -- 172.21.15.139:0/3178611168 <== osd.2 v2:172.21.15.139:6824/1043169956 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6cf80b8dc0 con 0x7f6cd8080800 2024-07-26T23:58:56.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cef7fe640 10 client.20302.objecter ms_dispatch 0x7f6cf8000b90 command_reply(tid 2: 0 ) 2024-07-26T23:58:56.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cef7fe640 10 client.20302.objecter _finish_command 2 = system:0 2024-07-26T23:58:56.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cef7fe640 20 client.20302.objecter put_session s=0x7f6cd8080130 osd=2 4 2024-07-26T23:58:56.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cef7fe640 15 client.20302.objecter _session_command_op_remove 2 2 2024-07-26T23:58:56.949 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:58:56.950 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:58:56.950 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:58:56.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 10 librados: watch_flush enter 2024-07-26T23:58:56.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 10 librados: watch_flush exit 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 10 client.20302.objecter close_session for osd.2 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f6cd8080800 msgr2=0x7f6cd8082ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.139:6824/1043169956,v1:172.21.15.139:6825/1043169956] conn(0x7f6cd8080800 0x7f6cd8082ca0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 20 client.20302.objecter put_session s=0x7f6cd8080130 osd=2 1 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 20 client.20302.objecter shutdown clearing up homeless session... 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 10 client.20302.objecter successfully canceled tick 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6cd8078a20 msgr2=0x7f6cd807aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f6cd8078a20 0x7f6cd807aee0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0036000 tx=0x7f6ce003c040 comp rx=0 tx=0).stop 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 10 monclient: shutdown 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 msgr2=0x7f6cf8143a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:56.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 --2- 172.21.15.139:0/3178611168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cf80b41c0 0x7f6cf8143a70 secure :-1 s=READY pgs=4940 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800cbe0 tx=0x7f6ce800a430 comp rx=0 tx=0).stop 2024-07-26T23:58:56.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cf77fe640 1 -- 172.21.15.139:0/3178611168 reap_dead start 2024-07-26T23:58:56.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 shutdown_connections 2024-07-26T23:58:56.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 >> 172.21.15.139:0/3178611168 conn(0x7f6cf80b62c0 msgr2=0x7f6cf814a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:56.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 shutdown_connections 2024-07-26T23:58:56.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 -- 172.21.15.139:0/3178611168 wait complete. 2024-07-26T23:58:56.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:56.934+0000 7f6cfd6dd640 1 librados: shutdown 2024-07-26T23:58:56.960 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-07-26T23:58:57.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.050+0000 7fa1fecb6640 10 monclient: get_monmap_and_config 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.050+0000 7fa1fecb6640 10 monclient: build_initial_monmap 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.050+0000 7fa1fecb6640 10 monclient: monmap: 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:57.051960+0000 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:57.051960+0000 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:57.066 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.067 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:57.067 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:57.067 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:57.067 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:57.067 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:57.067 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.069 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.051+0000 7fa1fecb6640 1 Processor -- start 2024-07-26T23:58:57.069 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.051+0000 7fa1fecb6640 1 -- start start 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.051+0000 7fa1fecb6640 10 monclient: init 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fa1f81470c0 0x7fa1f8140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f81470c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f8140e90 0x7fa1f80b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f8140e90 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1f80b4700 addr v1:172.21.15.139:6789/0 2024-07-26T23:58:57.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f8147ce0 con 0x7fa1f80b4700 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fa1f80acaf0 con 0x7fa1f81470c0 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f81477c0 con 0x7fa1f8140e90 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.052+0000 7fa1fecb6640 10 monclient(hunting): authenticate will time out at 2770.088392s 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1fca52640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fa1f81470c0 0x7fa1f8140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1f7fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f8140e90 0x7fa1f80b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1f7fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f8140e90 0x7fa1f80b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48446/0 (socket says 172.21.15.139:48446) 2024-07-26T23:58:57.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1f7fff640 1 -- 172.21.15.139:0/3847761221 learned_addr learned my addr 172.21.15.139:0/3847761221 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1f7fff640 10 monclient(hunting): get_auth_request con 0x7fa1f8140e90 auth_method 0 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1f7fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1fca52640 10 monclient(hunting): get_auth_request con 0x7fa1f81470c0 auth_method 0 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1fca52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1fca52640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.053+0000 7fa1fca52640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.054+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.054+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.072 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.054+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.054+0000 7fa1fca52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.054+0000 7fa1fca52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.073 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.054+0000 7fa1fca52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_done global_id 16841 payload 274 2024-07-26T23:58:57.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 1 -- 172.21.15.139:0/3847761221 >> v2:172.21.15.139:3300/0 conn(0x7fa1f81470c0 msgr2=0x7fa1f8140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:57.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 1 --2- 172.21.15.139:0/3847761221 >> v2:172.21.15.139:3300/0 conn(0x7fa1f81470c0 0x7fa1f8140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 1 -- 172.21.15.139:0/3847761221 >> v1:172.21.15.139:6789/0 conn(0x7fa1f80b4700 legacy=0x7fa1f81438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:57.074 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:57.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 1 monclient: found mon.noname-b 2024-07-26T23:58:57.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:57.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:58:57.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.055+0000 7fa1f7fff640 1 -- 172.21.15.139:0/3847761221 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f80b4b90 con 0x7fa1f8140e90 2024-07-26T23:58:57.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.058+0000 7fa1f7fff640 1 --2- 172.21.15.139:0/3847761221 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f8140e90 0x7fa1f80b41c0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009870 tx=0x7fa1e402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf8f68d5027d2f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.085 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 1 -- 172.21.15.139:0/3847761221 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40317c0 con 0x7fa1f8140e90 2024-07-26T23:58:57.085 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.085 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:57.086 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: dump: 2024-07-26T23:58:57.086 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.086 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.086 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.086 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:57.087 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 1 -- 172.21.15.139:0/3847761221 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e4037070 con 0x7fa1f8140e90 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1fecb6640 5 monclient: authenticate success, global_id 16841 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1fecb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1fecb6640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 1 -- 172.21.15.139:0/3847761221 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4031b80 con 0x7fa1f8140e90 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1fecb6640 4 set_mon_vals no callback set 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.062+0000 7fa1f77fe640 10 monclient: dump: 2024-07-26T23:58:57.088 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.089 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 10 monclient: shutdown 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 -- 172.21.15.139:0/3847761221 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f8140e90 msgr2=0x7fa1f80b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/3847761221 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f8140e90 0x7fa1f80b41c0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009870 tx=0x7fa1e402b860 comp rx=0 tx=0).stop 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 -- 172.21.15.139:0/3847761221 shutdown_connections 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/3847761221 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f8140e90 0x7fa1f80b41c0 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/3847761221 >> v2:172.21.15.139:3300/0 conn(0x7fa1f81470c0 0x7fa1f8140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 -- 172.21.15.139:0/3847761221 >> 172.21.15.139:0/3847761221 conn(0x7fa1f80b62c0 msgr2=0x7fa1f80b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 -- 172.21.15.139:0/3847761221 shutdown_connections 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.063+0000 7fa1fecb6640 1 -- 172.21.15.139:0/3847761221 wait complete. 2024-07-26T23:58:57.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.064+0000 7fa1fecb6640 10 monclient: build_initial_monmap 2024-07-26T23:58:57.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.064+0000 7fa1fecb6640 10 monclient: monmap: 2024-07-26T23:58:57.091 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:57.091 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:57.091 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:57.091 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:57.091 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:57.092 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.092 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:57.092 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:57.092 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:57.092 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.064+0000 7fa1fecb6640 1 librados: starting msgr at 2024-07-26T23:58:57.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.064+0000 7fa1fecb6640 1 librados: starting objecter 2024-07-26T23:58:57.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.064+0000 7fa1fecb6640 1 Processor -- start 2024-07-26T23:58:57.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 -- start start 2024-07-26T23:58:57.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 librados: setting wanted keys 2024-07-26T23:58:57.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 librados: calling monclient init 2024-07-26T23:58:57.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient: init 2024-07-26T23:58:57.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:57.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:58:57.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f80b4700 0x7fa1f81437d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f80b4700 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:57.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1f8140e90 0x7fa1f81ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f8140e90 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:57.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f81470c0 0x7fa1f81ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f81470c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f8074850 con 0x7fa1f8140e90 2024-07-26T23:58:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f80699f0 con 0x7fa1f81470c0 2024-07-26T23:58:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f8141410 con 0x7fa1f80b4700 2024-07-26T23:58:57.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:57.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.065+0000 7fa1fecb6640 10 monclient(hunting): authenticate will time out at 2770.101734s 2024-07-26T23:58:57.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fca52640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f80b4700 0x7fa1f81437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1f7fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1f8140e90 0x7fa1f81ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fd253640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f81470c0 0x7fa1f81ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fca52640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f80b4700 0x7fa1f81437d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48450/0 (socket says 172.21.15.139:48450) 2024-07-26T23:58:57.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1f7fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1f8140e90 0x7fa1f81ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36350/0 (socket says 172.21.15.139:36350) 2024-07-26T23:58:57.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fca52640 1 -- 172.21.15.139:0/720573984 learned_addr learned my addr 172.21.15.139:0/720573984 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:57.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fca52640 10 monclient(hunting): get_auth_request con 0x7fa1f80b4700 auth_method 0 2024-07-26T23:58:57.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fca52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fca52640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fca52640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1f7fff640 10 monclient(hunting): get_auth_request con 0x7fa1f8140e90 auth_method 0 2024-07-26T23:58:57.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1f7fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fd253640 10 monclient(hunting): get_auth_request con 0x7fa1f81470c0 auth_method 0 2024-07-26T23:58:57.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fd253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fd253640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.066+0000 7fa1fd253640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1fca52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1fca52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1fca52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1fd253640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.101 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1fd253640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.102 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.067+0000 7fa1fd253640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.106 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 10 monclient(hunting): handle_auth_done global_id 15564 payload 824 2024-07-26T23:58:57.106 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 1 -- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f80b4700 msgr2=0x7fa1f81437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.106 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f80b4700 0x7fa1f81437d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.106 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 1 -- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f81470c0 msgr2=0x7fa1f81ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.107 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f81470c0 0x7fa1f81ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.107 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:57.107 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 1 monclient: found mon.noname-a 2024-07-26T23:58:57.107 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:57.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:57.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1f7fff640 1 -- 172.21.15.139:0/720573984 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f81d0a00 con 0x7fa1f8140e90 2024-07-26T23:58:57.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.071+0000 7fa1fca52640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f80b4700 0x7fa1f81437d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T23:58:57.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.074+0000 7fa1f7fff640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1f8140e90 0x7fa1f81ccec0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4033000 tx=0x7fa1e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.074+0000 7fa1f57fa640 10 client.?.objecter ms_handle_connect 0x7fa1f8140e90 2024-07-26T23:58:57.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.074+0000 7fa1f57fa640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:57.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 1 -- 172.21.15.139:0/720573984 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4032650 con 0x7fa1f8140e90 2024-07-26T23:58:57.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:57.109 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: dump: 2024-07-26T23:58:57.109 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.110 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.110 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.110 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.110 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.110 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.111 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.111 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.111 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.111 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: _finish_auth 0 2024-07-26T23:58:57.111 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: _check_auth_tickets 2024-07-26T23:58:57.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:57.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 1 -- 172.21.15.139:0/720573984 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e4032830 con 0x7fa1f8140e90 2024-07-26T23:58:57.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:57.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 1 -- 172.21.15.139:0/720573984 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4032c20 con 0x7fa1f8140e90 2024-07-26T23:58:57.112 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:57.113 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f57fa640 10 monclient: dump: 2024-07-26T23:58:57.113 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.113 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.113 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.113 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.114 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.114 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.114 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.114 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 5 monclient: authenticate success, global_id 15564 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 10 monclient: _renew_subs 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f81ce870 con 0x7fa1f8140e90 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1f6ffd640 4 set_mon_vals no callback set 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 10 client.15564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:57.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 10 monclient: _renew_subs 2024-07-26T23:58:57.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:57.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f81cf3b0 con 0x7fa1f8140e90 2024-07-26T23:58:57.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.078+0000 7fa1fecb6640 1 librados: init done 2024-07-26T23:58:57.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.079+0000 7fa1fecb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:57.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.079+0000 7fa1fecb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:57.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.079+0000 7fa1fecb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:57.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.079+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1f80b41c0 con 0x7fa1f8140e90 2024-07-26T23:58:57.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.101+0000 7fa1f57fa640 1 -- 172.21.15.139:0/720573984 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fa1e404c060 con 0x7fa1f8140e90 2024-07-26T23:58:57.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.101+0000 7fa1f57fa640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa1dc0789a0 0x7fa1dc07ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.102+0000 7fa1fca52640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa1dc0789a0 0x7fa1dc07ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.102+0000 7fa1fca52640 10 monclient: get_auth_request con 0x7fa1dc0789a0 auth_method 0 2024-07-26T23:58:57.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.102+0000 7fa1f57fa640 10 client.15564.objecter ms_handle_connect 0x7fa1dc0789a0 2024-07-26T23:58:57.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.102+0000 7fa1fca52640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa1dc0789a0 0x7fa1dc07ae60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009f50 tx=0x7fa1ec009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.105+0000 7fa1f57fa640 1 -- 172.21.15.139:0/720573984 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7fa1e4032dc0 con 0x7fa1f8140e90 2024-07-26T23:58:57.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.105+0000 7fa1f57fa640 10 client.15564.objecter ms_dispatch 0x7fa1f8000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:57.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.105+0000 7fa1f57fa640 3 client.15564.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:57.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.105+0000 7fa1f57fa640 3 client.15564.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:57.120 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.105+0000 7fa1f57fa640 20 client.15564.objecter dump_active .. 0 homeless 2024-07-26T23:58:57.387 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.371+0000 7fa1f57fa640 1 -- 172.21.15.139:0/720573984 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fa1e4040070 con 0x7fa1f8140e90 2024-07-26T23:58:57.387 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.371+0000 7fa1f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:57.387 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.371+0000 7fa1f57fa640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:57.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.561+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fa1f813c3a0 con 0x7fa1dc0789a0 2024-07-26T23:58:57.577 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.561+0000 7fa1f57fa640 1 -- 172.21.15.139:0/720573984 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa1f813c3a0 con 0x7fa1dc0789a0 2024-07-26T23:58:57.577 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 2 to deep-scrub 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 10 librados: watch_flush enter 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 10 librados: watch_flush exit 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 20 client.15564.objecter shutdown clearing up homeless session... 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 10 client.15564.objecter successfully canceled tick 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa1dc0789a0 msgr2=0x7fa1dc07ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa1dc0789a0 0x7fa1dc07ae60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009f50 tx=0x7fa1ec009190 comp rx=0 tx=0).stop 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 10 monclient: shutdown 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1f8140e90 msgr2=0x7fa1f81ccec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1f8140e90 0x7fa1f81ccec0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4033000 tx=0x7fa1e40046c0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 shutdown_connections 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fa1dc0789a0 0x7fa1dc07ae60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f81470c0 0x7fa1f81ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1f8140e90 0x7fa1f81ccec0 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 --2- 172.21.15.139:0/720573984 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fa1f80b4700 0x7fa1f81437d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 >> 172.21.15.139:0/720573984 conn(0x7fa1f80b62c0 msgr2=0x7fa1f8068c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:57.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 shutdown_connections 2024-07-26T23:58:57.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 -- 172.21.15.139:0/720573984 wait complete. 2024-07-26T23:58:57.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.564+0000 7fa1fecb6640 1 librados: shutdown 2024-07-26T23:58:57.590 INFO:tasks.ceph:Scrubbing osd.3 2024-07-26T23:58:57.590 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.697+0000 7f998107c640 10 monclient: get_monmap_and_config 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.697+0000 7f998107c640 10 monclient: build_initial_monmap 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.697+0000 7f998107c640 10 monclient: monmap: 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:57.698264+0000 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:57.698264+0000 2024-07-26T23:58:57.712 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:57.713 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.713 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:57.713 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:57.713 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:57.713 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:57.713 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:57.713 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.697+0000 7f998107c640 1 Processor -- start 2024-07-26T23:58:57.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.697+0000 7f998107c640 1 -- start start 2024-07-26T23:58:57.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.697+0000 7f998107c640 10 monclient: init 2024-07-26T23:58:57.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:57.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f997c06e720 0x7f997c139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient(hunting): picked mon.noname-c con 0x7f997c06e720 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f997c143d00 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f997c144cd0 0x7f997c145100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient(hunting): picked mon.noname-a con 0x7f997c144cd0 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f997c0ac810 con 0x7f997c143d00 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f997c06eb00 con 0x7f997c144cd0 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f997c144710 con 0x7f997c06e720 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.698+0000 7f998107c640 10 monclient(hunting): authenticate will time out at 2770.734485s 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f997c144cd0 0x7f997c145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997ad9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f997c06e720 0x7f997c139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f997c144cd0 0x7f997c145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36356/0 (socket says 172.21.15.139:36356) 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 1 -- 172.21.15.139:0/2896091634 learned_addr learned my addr 172.21.15.139:0/2896091634 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 10 monclient(hunting): get_auth_request con 0x7f997c144cd0 auth_method 0 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997ad9d640 10 monclient(hunting): get_auth_request con 0x7f997c06e720 auth_method 0 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997ad9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.699+0000 7f997b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.700+0000 7f997ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.700+0000 7f997ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.700+0000 7f997ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 10 monclient(hunting): handle_auth_done global_id 15567 payload 274 2024-07-26T23:58:57.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 1 -- 172.21.15.139:0/2896091634 >> v2:172.21.15.177:3300/0 conn(0x7f997c06e720 msgr2=0x7f997c139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:57.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 1 --2- 172.21.15.139:0/2896091634 >> v2:172.21.15.177:3300/0 conn(0x7f997c06e720 0x7f997c139bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 1 -- 172.21.15.139:0/2896091634 >> v1:172.21.15.177:6789/0 conn(0x7f997c143d00 legacy=0x7f997c13a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 10 monclient: _finish_hunting 0 2024-07-26T23:58:57.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 1 monclient: found mon.noname-a 2024-07-26T23:58:57.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:57.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:57.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.701+0000 7f997b59e640 1 -- 172.21.15.139:0/2896091634 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c0acaf0 con 0x7f997c144cd0 2024-07-26T23:58:57.719 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.703+0000 7f997b59e640 1 --2- 172.21.15.139:0/2896091634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c144cd0 0x7f997c145100 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f997c0b7360 tx=0x7f996c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7296c36c45c8808 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 1 -- 172.21.15.139:0/2896091634 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c02cb20 con 0x7f997c144cd0 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: dump: 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.723 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: _finish_auth 0 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 1 -- 172.21.15.139:0/2896091634 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f996c02ccc0 con 0x7f997c144cd0 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 5 monclient: authenticate success, global_id 15567 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:58:57.724 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 1 -- 172.21.15.139:0/2896091634 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c034610 con 0x7f997c144cd0 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 4 set_mon_vals no callback set 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f9979d9b640 10 monclient: dump: 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.725 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 10 monclient: shutdown 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 -- 172.21.15.139:0/2896091634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c144cd0 msgr2=0x7f997c145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 --2- 172.21.15.139:0/2896091634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c144cd0 0x7f997c145100 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f997c0b7360 tx=0x7f996c02bf80 comp rx=0 tx=0).stop 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 -- 172.21.15.139:0/2896091634 shutdown_connections 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 --2- 172.21.15.139:0/2896091634 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c144cd0 0x7f997c145100 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 --2- 172.21.15.139:0/2896091634 >> v2:172.21.15.177:3300/0 conn(0x7f997c06e720 0x7f997c139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 -- 172.21.15.139:0/2896091634 >> 172.21.15.139:0/2896091634 conn(0x7f997c0b9c10 msgr2=0x7f997c0ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 -- 172.21.15.139:0/2896091634 shutdown_connections 2024-07-26T23:58:57.726 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.707+0000 7f998107c640 1 -- 172.21.15.139:0/2896091634 wait complete. 2024-07-26T23:58:57.729 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.708+0000 7f998107c640 10 monclient: build_initial_monmap 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.708+0000 7f998107c640 10 monclient: monmap: 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.730 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.709+0000 7f998107c640 1 librados: starting msgr at 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.709+0000 7f998107c640 1 librados: starting objecter 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.709+0000 7f998107c640 1 Processor -- start 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.709+0000 7f998107c640 1 -- start start 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.709+0000 7f998107c640 1 librados: setting wanted keys 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.709+0000 7f998107c640 1 librados: calling monclient init 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.709+0000 7f998107c640 10 monclient: init 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c06e720 0x7f997c146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): picked mon.noname-a con 0x7f997c06e720 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f997c143d00 0x7f997c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): picked mon.noname-c con 0x7f997c143d00 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:57.731 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c144cd0 0x7f997c1dc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): picked mon.noname-b con 0x7f997c144cd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f997c145640 con 0x7f997c06e720 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f997c141390 con 0x7f997c144cd0 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f997c1457c0 con 0x7f997c143d00 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f998107c640 10 monclient(hunting): authenticate will time out at 2770.746138s 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997a59c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f997c143d00 0x7f997c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997b59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c144cd0 0x7f997c1dc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997b59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c144cd0 0x7f997c1dc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:46954/0 (socket says 172.21.15.139:46954) 2024-07-26T23:58:57.732 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997b59e640 1 -- 172.21.15.139:0/3889098719 learned_addr learned my addr 172.21.15.139:0/3889098719 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997b59e640 10 monclient(hunting): get_auth_request con 0x7f997c144cd0 auth_method 0 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997b59e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.710+0000 7f997b59e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997a59c640 10 monclient(hunting): get_auth_request con 0x7f997c143d00 auth_method 0 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997a59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997a59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997ad9d640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c06e720 0x7f997c146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.733 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997ad9d640 10 monclient(hunting): get_auth_request con 0x7f997c06e720 auth_method 0 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997ad9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.711+0000 7f997ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.712+0000 7f997ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.712+0000 7f997ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.712+0000 7f997ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.713+0000 7f997b59e640 10 monclient(hunting): handle_auth_done global_id 20311 payload 824 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.713+0000 7f997b59e640 1 -- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f997c143d00 msgr2=0x7f997c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f997c143d00 0x7f997c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 1 -- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c06e720 msgr2=0x7f997c146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.734 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f997c06e720 0x7f997c146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.735 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 10 monclient: _finish_hunting 0 2024-07-26T23:58:57.735 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 1 monclient: found mon.noname-b 2024-07-26T23:58:57.735 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:57.735 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:57.735 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.714+0000 7f997b59e640 1 -- 172.21.15.139:0/3889098719 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c135820 con 0x7f997c144cd0 2024-07-26T23:58:57.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.716+0000 7f997b59e640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c144cd0 0x7f997c1dc550 secure :-1 s=READY pgs=4942 cs=0 l=1 rev1=1 crypto rx=0x7f996c036040 tx=0x7f996c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.716+0000 7f99637fe640 10 client.?.objecter ms_handle_connect 0x7f997c144cd0 2024-07-26T23:58:57.736 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.716+0000 7f99637fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c043020 con 0x7f997c144cd0 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: dump: 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.737 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 5 monclient: authenticate success, global_id 20311 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f996c03e450 con 0x7f997c144cd0 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c03e840 con 0x7f997c144cd0 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 10 monclient: _renew_subs 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c1308a0 con 0x7f997c144cd0 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.738 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:57.739 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f99637fe640 10 monclient: dump: 2024-07-26T23:58:57.739 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.739 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.739 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.741 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f997959a640 4 set_mon_vals no callback set 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 10 client.20311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 10 monclient: _renew_subs 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c13e2f0 con 0x7f997c144cd0 2024-07-26T23:58:57.742 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.719+0000 7f998107c640 1 librados: init done 2024-07-26T23:58:57.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.720+0000 7f998107c640 10 client.20311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:57.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.720+0000 7f998107c640 15 client.20311.objecter _session_command_op_assign -1 1 2024-07-26T23:58:57.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.720+0000 7f998107c640 20 client.20311.objecter _get_session osd=-1 returning homeless 2024-07-26T23:58:57.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.720+0000 7f998107c640 10 client.20311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:57.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.720+0000 7f998107c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:57.743 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.720+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f997c050bb0 con 0x7f997c144cd0 2024-07-26T23:58:57.756 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.741+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f996c046020 con 0x7f997c144cd0 2024-07-26T23:58:57.756 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.741+0000 7f99637fe640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f99680789f0 0x7f996807aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.756 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.741+0000 7f997ad9d640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f99680789f0 0x7f996807aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.756 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.741+0000 7f997ad9d640 10 monclient: get_auth_request con 0x7f99680789f0 auth_method 0 2024-07-26T23:58:57.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.742+0000 7f99637fe640 10 client.20311.objecter ms_handle_connect 0x7f99680789f0 2024-07-26T23:58:57.757 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.742+0000 7f997ad9d640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f99680789f0 0x7f996807aeb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9974009b40 tx=0x7f9974009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.760 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f996c052d30 con 0x7f997c144cd0 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 10 client.20311.objecter ms_dispatch 0x7f997c000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 3 client.20311.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 3 client.20311.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 10 client.20311.objecter checking command 1 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f99680807d0 0x7f9968082c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 20 client.20311.objecter _get_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 20 client.20311.objecter put_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 15 client.20311.objecter _session_command_op_remove -1 1 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 20 client.20311.objecter _get_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 20 client.20311.objecter get_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 15 client.20311.objecter _session_command_op_assign 3 1 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 20 client.20311.objecter put_session s=0x7f9968080100 osd=3 4 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 10 client.20311.objecter _send_command 1 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9968083340 con 0x7f99680807d0 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 20 client.20311.objecter dump_active .. 0 homeless 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=970) ==== 24+0+0 (secure 0 0 0) 0x7f996c080610 con 0x7f997c144cd0 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f99637fe640 10 monclient: handle_get_version_reply finishing 1 version 970 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.745+0000 7f997a59c640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f99680807d0 0x7f9968082c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.746+0000 7f997a59c640 10 monclient: get_auth_request con 0x7f99680807d0 auth_method 0 2024-07-26T23:58:57.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.748+0000 7f99637fe640 10 client.20311.objecter ms_handle_connect 0x7f99680807d0 2024-07-26T23:58:57.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.748+0000 7f997a59c640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f99680807d0 0x7f9968082c70 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.771+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== osd.3 v2:172.21.15.139:6816/4033735752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9968083340 con 0x7f99680807d0 2024-07-26T23:58:57.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.771+0000 7f99637fe640 10 client.20311.objecter ms_dispatch 0x7f997c000b90 command_reply(tid 1: 0 ) 2024-07-26T23:58:57.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.771+0000 7f99637fe640 10 client.20311.objecter _finish_command 1 = system:0 2024-07-26T23:58:57.786 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.771+0000 7f99637fe640 20 client.20311.objecter put_session s=0x7f9968080100 osd=3 4 2024-07-26T23:58:57.787 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.771+0000 7f99637fe640 15 client.20311.objecter _session_command_op_remove 3 1 2024-07-26T23:58:57.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 10 client.20311.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:58:57.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 15 client.20311.objecter _session_command_op_assign -1 2 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 20 client.20311.objecter _get_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 20 client.20311.objecter put_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 20 client.20311.objecter _get_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 15 client.20311.objecter _session_command_op_remove -1 2 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 20 client.20311.objecter get_session s=0x7f9968080100 osd=3 3 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 15 client.20311.objecter _session_command_op_assign 3 2 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 20 client.20311.objecter put_session s=0x7f9968080100 osd=3 4 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 10 client.20311.objecter _send_command 2 2024-07-26T23:58:57.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.784+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 --> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f997c145100 con 0x7f99680807d0 2024-07-26T23:58:57.812 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f99637fe640 1 -- 172.21.15.139:0/3889098719 <== osd.3 v2:172.21.15.139:6816/4033735752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f997c145100 con 0x7f99680807d0 2024-07-26T23:58:57.812 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f99637fe640 10 client.20311.objecter ms_dispatch 0x7f997c000b90 command_reply(tid 2: 0 ) 2024-07-26T23:58:57.812 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f99637fe640 10 client.20311.objecter _finish_command 2 = system:0 2024-07-26T23:58:57.812 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f99637fe640 20 client.20311.objecter put_session s=0x7f9968080100 osd=3 4 2024-07-26T23:58:57.812 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f99637fe640 15 client.20311.objecter _session_command_op_remove 3 2 2024-07-26T23:58:57.812 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:58:57.813 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:58:57.813 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:58:57.814 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 10 librados: watch_flush enter 2024-07-26T23:58:57.814 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 10 librados: watch_flush exit 2024-07-26T23:58:57.814 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 10 client.20311.objecter close_session for osd.3 2024-07-26T23:58:57.814 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f99680807d0 msgr2=0x7f9968082c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.814 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.139:6816/4033735752,v1:172.21.15.139:6817/4033735752] conn(0x7f99680807d0 0x7f9968082c70 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 20 client.20311.objecter put_session s=0x7f9968080100 osd=3 1 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 20 client.20311.objecter shutdown clearing up homeless session... 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.797+0000 7f998107c640 10 client.20311.objecter successfully canceled tick 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f99680789f0 msgr2=0x7f996807aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f99680789f0 0x7f996807aeb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9974009b40 tx=0x7f9974009190 comp rx=0 tx=0).stop 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 10 monclient: shutdown 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c144cd0 msgr2=0x7f997c1dc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 --2- 172.21.15.139:0/3889098719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c144cd0 0x7f997c1dc550 secure :-1 s=READY pgs=4942 cs=0 l=1 rev1=1 crypto rx=0x7f996c036040 tx=0x7f996c004a40 comp rx=0 tx=0).stop 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f997b59e640 1 -- 172.21.15.139:0/3889098719 reap_dead start 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 shutdown_connections 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 >> 172.21.15.139:0/3889098719 conn(0x7f997c0b9c10 msgr2=0x7f997c1476a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:57.815 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 shutdown_connections 2024-07-26T23:58:57.816 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 -- 172.21.15.139:0/3889098719 wait complete. 2024-07-26T23:58:57.816 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.798+0000 7f998107c640 1 librados: shutdown 2024-07-26T23:58:57.824 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-07-26T23:58:57.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.908+0000 7f143ce65640 10 monclient: get_monmap_and_config 2024-07-26T23:58:57.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.908+0000 7f143ce65640 10 monclient: build_initial_monmap 2024-07-26T23:58:57.923 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.908+0000 7f143ce65640 10 monclient: monmap: 2024-07-26T23:58:57.923 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:57.909397+0000 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:57.909397+0000 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:57.924 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.926 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 Processor -- start 2024-07-26T23:58:57.926 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 -- start start 2024-07-26T23:58:57.926 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient: init 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f14380b2660 0x7f1438074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): picked mon.noname-b con 0x7f14380b2660 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f14380b2a40 0x7f14380748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): picked mon.noname-c con 0x7f14380b2a40 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f1438143c60 0x7f1438144110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): picked mon.noname-a con 0x7f1438143c60 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f14380af400 con 0x7f1438143c60 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f14380af6e0 con 0x7f14380b2a40 2024-07-26T23:58:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f14380ad560 con 0x7f14380b2660 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.909+0000 7f143ce65640 10 monclient(hunting): authenticate will time out at 2770.945843s 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f143659c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f14380b2660 0x7f1438074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f1436d9d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f1438143c60 0x7f1438144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f142ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f14380b2a40 0x7f14380748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f143659c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f14380b2660 0x7f1438074370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48474/0 (socket says 172.21.15.139:48474) 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f143659c640 1 -- 172.21.15.139:0/3686463077 learned_addr learned my addr 172.21.15.139:0/3686463077 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f143659c640 10 monclient(hunting): get_auth_request con 0x7f14380b2660 auth_method 0 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f143659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f143659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f143659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f142ffff640 10 monclient(hunting): get_auth_request con 0x7f14380b2a40 auth_method 0 2024-07-26T23:58:57.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f142ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f142ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f142ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f1436d9d640 10 monclient(hunting): get_auth_request con 0x7f1438143c60 auth_method 0 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f1436d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f1436d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.910+0000 7f1436d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f143659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f143659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f143659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f142ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f142ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f142ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f1436d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f1436d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.911+0000 7f1436d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.930 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.912+0000 7f142ffff640 10 monclient(hunting): handle_auth_done global_id 20314 payload 274 2024-07-26T23:58:57.930 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.912+0000 7f142ffff640 1 -- 172.21.15.139:0/3686463077 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f14380b2660 msgr2=0x7f1438074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:57.930 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.912+0000 7f142ffff640 1 --2- 172.21.15.139:0/3686463077 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f14380b2660 0x7f1438074370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f142ffff640 1 -- 172.21.15.139:0/3686463077 >> v2:172.21.15.139:3300/0 conn(0x7f1438143c60 msgr2=0x7f1438144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f142ffff640 1 --2- 172.21.15.139:0/3686463077 >> v2:172.21.15.139:3300/0 conn(0x7f1438143c60 0x7f1438144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f142ffff640 10 monclient: _finish_hunting 0 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f142ffff640 1 monclient: found mon.noname-c 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f142ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f142ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f142ffff640 1 -- 172.21.15.139:0/3686463077 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14381447a0 con 0x7f14380b2a40 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.913+0000 7f143659c640 1 --2- 172.21.15.139:0/3686463077 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f14380b2660 0x7f1438074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T23:58:57.931 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.915+0000 7f142ffff640 1 --2- 172.21.15.139:0/3686463077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14380b2a40 0x7f14380748b0 secure :-1 s=READY pgs=4943 cs=0 l=1 rev1=1 crypto rx=0x7f143806ded0 tx=0x7f142002def0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f09e8dbfb061c0ce server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 1 -- 172.21.15.139:0/3686463077 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142003b070 con 0x7f14380b2a40 2024-07-26T23:58:57.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: dump: 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: _finish_auth 0 2024-07-26T23:58:57.934 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 1 -- 172.21.15.139:0/3686463077 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f142002eca0 con 0x7f14380b2a40 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 1 -- 172.21.15.139:0/3686463077 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420036660 con 0x7f14380b2a40 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f143ce65640 5 monclient: authenticate success, global_id 20314 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f143ce65640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f143ce65640 4 set_mon_vals no callback set 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.918+0000 7f1435d9b640 10 monclient: dump: 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.935 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.936 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.936 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.936 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.936 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.936 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 10 monclient: shutdown 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 -- 172.21.15.139:0/3686463077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14380b2a40 msgr2=0x7f14380748b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 --2- 172.21.15.139:0/3686463077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14380b2a40 0x7f14380748b0 secure :-1 s=READY pgs=4943 cs=0 l=1 rev1=1 crypto rx=0x7f143806ded0 tx=0x7f142002def0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 -- 172.21.15.139:0/3686463077 shutdown_connections 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 --2- 172.21.15.139:0/3686463077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14380b2a40 0x7f14380748b0 unknown :-1 s=CLOSED pgs=4943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 --2- 172.21.15.139:0/3686463077 >> v2:172.21.15.139:3300/0 conn(0x7f1438143c60 0x7f1438144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 --2- 172.21.15.139:0/3686463077 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f14380b2660 0x7f1438074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 -- 172.21.15.139:0/3686463077 >> 172.21.15.139:0/3686463077 conn(0x7f14380b5b40 msgr2=0x7f14380b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 -- 172.21.15.139:0/3686463077 shutdown_connections 2024-07-26T23:58:57.939 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.919+0000 7f143ce65640 1 -- 172.21.15.139:0/3686463077 wait complete. 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 10 monclient: build_initial_monmap 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 10 monclient: monmap: 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 1 librados: starting msgr at 2024-07-26T23:58:57.940 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 1 librados: starting objecter 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 1 Processor -- start 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 1 -- start start 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 1 librados: setting wanted keys 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 1 librados: calling monclient init 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.920+0000 7f143ce65640 10 monclient: init 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient: _add_conns ranks=[0,1,2] 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14380b2660 0x7f1438073710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): picked mon.noname-a con 0x7f14380b2660 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1438143c60 0x7f143813b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): picked mon.noname-b con 0x7f1438143c60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f143813c760 0x7f14381dc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): picked mon.noname-c con 0x7f143813c760 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.941 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f14380ae3a0 con 0x7f14380b2660 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14380ae220 con 0x7f1438143c60 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f14380b0e60 con 0x7f143813c760 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143ce65640 10 monclient(hunting): authenticate will time out at 2770.957357s 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f1436d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f143813c760 0x7f14381dc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f143659c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14380b2660 0x7f1438073710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.921+0000 7f142ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1438143c60 0x7f143813b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f143813c760 0x7f14381dc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48490/0 (socket says 172.21.15.139:48490) 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 1 -- 172.21.15.139:0/2172901293 learned_addr learned my addr 172.21.15.139:0/2172901293 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 10 monclient(hunting): get_auth_request con 0x7f143813c760 auth_method 0 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f143659c640 10 monclient(hunting): get_auth_request con 0x7f14380b2660 auth_method 0 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f143659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f143659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f143659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f142ffff640 10 monclient(hunting): get_auth_request con 0x7f1438143c60 auth_method 0 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f142ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f142ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:57.943 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f142ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f1436d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f142ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f142ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f142ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f143659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f143659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:57.944 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.922+0000 7f143659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:57.947 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 10 monclient(hunting): handle_auth_done global_id 16853 payload 824 2024-07-26T23:58:57.947 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 1 -- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1438143c60 msgr2=0x7f143813b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.947 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1438143c60 0x7f143813b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.947 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 1 -- 172.21.15.139:0/2172901293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14380b2660 msgr2=0x7f1438073710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14380b2660 0x7f1438073710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 10 monclient: _finish_hunting 0 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 1 monclient: found mon.noname-c 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f1436d9d640 1 -- 172.21.15.139:0/2172901293 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143813d8e0 con 0x7f143813c760 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.925+0000 7f142ffff640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1438143c60 0x7f143813b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.927+0000 7f1436d9d640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f143813c760 0x7f14381dc0d0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f143000efc0 tx=0x7f143000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.927+0000 7f142effd640 10 client.?.objecter ms_handle_connect 0x7f143813c760 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.927+0000 7f142effd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f142effd640 1 -- 172.21.15.139:0/2172901293 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1430018020 con 0x7f143813c760 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f142effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:57.948 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f142effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f142effd640 10 monclient: dump: 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f142effd640 10 monclient: _finish_auth 0 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f142effd640 10 monclient: _check_auth_tickets 2024-07-26T23:58:57.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f142effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 5 monclient: authenticate success, global_id 16853 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 10 monclient: _renew_subs 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f143813dbc0 con 0x7f143813c760 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 10 client.16853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 10 monclient: _renew_subs 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14381dc610 con 0x7f143813c760 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.929+0000 7f143ce65640 1 librados: init done 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.930+0000 7f143ce65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.930+0000 7f143ce65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.930+0000 7f143ce65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:58:57.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:57.930+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1438144110 con 0x7f143813c760 2024-07-26T23:58:58.154 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.139+0000 7f142effd640 1 -- 172.21.15.139:0/2172901293 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1430014480 con 0x7f143813c760 2024-07-26T23:58:58.154 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.139+0000 7f142effd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:58.154 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.139+0000 7f143559a640 4 set_mon_vals no callback set 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.197+0000 7f142effd640 1 -- 172.21.15.139:0/2172901293 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1430014850 con 0x7f143813c760 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.197+0000 7f142effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.197+0000 7f142effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.197+0000 7f142effd640 10 monclient: dump: 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.212 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.213 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.213 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.213 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.213 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.213 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.213 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.197+0000 7f142effd640 1 -- 172.21.15.139:0/2172901293 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f1430005d10 con 0x7f143813c760 2024-07-26T23:58:58.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.198+0000 7f142effd640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f14180789a0 0x7f141807ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.198+0000 7f143659c640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f14180789a0 0x7f141807ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.198+0000 7f143659c640 10 monclient: get_auth_request con 0x7f14180789a0 auth_method 0 2024-07-26T23:58:58.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.198+0000 7f142effd640 10 client.16853.objecter ms_handle_connect 0x7f14180789a0 2024-07-26T23:58:58.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.198+0000 7f143659c640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f14180789a0 0x7f141807ae60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f14280096d0 tx=0x7f1428009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.213+0000 7f142effd640 1 -- 172.21.15.139:0/2172901293 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f1430097e90 con 0x7f143813c760 2024-07-26T23:58:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.213+0000 7f142effd640 10 client.16853.objecter ms_dispatch 0x7f1438000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.213+0000 7f142effd640 3 client.16853.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.213+0000 7f142effd640 3 client.16853.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.214+0000 7f142effd640 20 client.16853.objecter dump_active .. 0 homeless 2024-07-26T23:58:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.214+0000 7f142effd640 1 -- 172.21.15.139:0/2172901293 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f1430022070 con 0x7f143813c760 2024-07-26T23:58:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.214+0000 7f142effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:58.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.214+0000 7f142effd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:58.428 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.413+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f143806b560 con 0x7f14180789a0 2024-07-26T23:58:58.429 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.413+0000 7f142effd640 1 -- 172.21.15.139:0/2172901293 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f143806b560 con 0x7f14180789a0 2024-07-26T23:58:58.429 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 3 to deep-scrub 2024-07-26T23:58:58.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 10 librados: watch_flush enter 2024-07-26T23:58:58.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 10 librados: watch_flush exit 2024-07-26T23:58:58.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 20 client.16853.objecter shutdown clearing up homeless session... 2024-07-26T23:58:58.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 10 client.16853.objecter successfully canceled tick 2024-07-26T23:58:58.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f14180789a0 msgr2=0x7f141807ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f14180789a0 0x7f141807ae60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f14280096d0 tx=0x7f1428009240 comp rx=0 tx=0).stop 2024-07-26T23:58:58.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 10 monclient: shutdown 2024-07-26T23:58:58.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:58.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f143813c760 msgr2=0x7f14381dc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f143813c760 0x7f14381dc0d0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f143000efc0 tx=0x7f143000c490 comp rx=0 tx=0).stop 2024-07-26T23:58:58.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 shutdown_connections 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f14180789a0 0x7f141807ae60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f143813c760 0x7f14381dc0d0 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1438143c60 0x7f143813b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 --2- 172.21.15.139:0/2172901293 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14380b2660 0x7f1438073710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 >> 172.21.15.139:0/2172901293 conn(0x7f14380b5b40 msgr2=0x7f143813fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 shutdown_connections 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 -- 172.21.15.139:0/2172901293 wait complete. 2024-07-26T23:58:58.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.416+0000 7f143ce65640 1 librados: shutdown 2024-07-26T23:58:58.443 INFO:tasks.ceph:Scrubbing osd.4 2024-07-26T23:58:58.444 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:58:58.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.541+0000 7f0f86068640 10 monclient: get_monmap_and_config 2024-07-26T23:58:58.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.541+0000 7f0f86068640 10 monclient: build_initial_monmap 2024-07-26T23:58:58.556 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.541+0000 7f0f86068640 10 monclient: monmap: 2024-07-26T23:58:58.556 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:58.542125+0000 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:58.542125+0000 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:58.557 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.565 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.541+0000 7f0f86068640 1 Processor -- start 2024-07-26T23:58:58.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.541+0000 7f0f86068640 1 -- start start 2024-07-26T23:58:58.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.541+0000 7f0f86068640 10 monclient: init 2024-07-26T23:58:58.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:58.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f801470c0 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f800b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient(hunting): picked mon.noname-b con 0x7f0f800b41c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0f8014c1d0 0x7f0f8014e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient(hunting): picked mon.noname-c con 0x7f0f8014c1d0 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f80147ce0 con 0x7f0f801470c0 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0f800acaf0 con 0x7f0f8014c1d0 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f0f801477c0 con 0x7f0f800b41c0 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.542+0000 7f0f86068640 10 monclient(hunting): authenticate will time out at 2771.578589s 2024-07-26T23:58:58.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f76ffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f800b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f7ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0f8014c1d0 0x7f0f8014e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f76ffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f800b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48498/0 (socket says 172.21.15.139:48498) 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f76ffd640 1 -- 172.21.15.139:0/2157236686 learned_addr learned my addr 172.21.15.139:0/2157236686 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f76ffd640 10 monclient(hunting): get_auth_request con 0x7f0f800b41c0 auth_method 0 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f76ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f7ffff640 10 monclient(hunting): get_auth_request con 0x7f0f8014c1d0 auth_method 0 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f7ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.543+0000 7f0f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.544+0000 7f0f76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.544+0000 7f0f76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.544+0000 7f0f76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.544+0000 7f0f7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.544+0000 7f0f7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.544+0000 7f0f7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 10 monclient(hunting): handle_auth_done global_id 16856 payload 274 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 1 -- 172.21.15.139:0/2157236686 >> v2:172.21.15.177:3300/0 conn(0x7f0f8014c1d0 msgr2=0x7f0f8014e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 1 --2- 172.21.15.139:0/2157236686 >> v2:172.21.15.177:3300/0 conn(0x7f0f8014c1d0 0x7f0f8014e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 1 -- 172.21.15.139:0/2157236686 >> v1:172.21.15.177:6789/0 conn(0x7f0f801470c0 legacy=0x7f0f80140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:58.570 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:58:58.570 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 1 monclient: found mon.noname-b 2024-07-26T23:58:58.570 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:58.570 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:58:58.570 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.545+0000 7f0f76ffd640 1 -- 172.21.15.139:0/2157236686 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f801474a0 con 0x7f0f800b41c0 2024-07-26T23:58:58.570 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.547+0000 7f0f76ffd640 1 --2- 172.21.15.139:0/2157236686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f800b45e0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f0f680098a0 tx=0x7f0f6802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb91f4b3c86c5a28 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 1 -- 172.21.15.139:0/2157236686 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f680315a0 con 0x7f0f800b41c0 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: dump: 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.575 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: _finish_auth 0 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: _check_auth_tickets 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 1 -- 172.21.15.139:0/2157236686 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f68037070 con 0x7f0f800b41c0 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 1 -- 172.21.15.139:0/2157236686 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f68031960 con 0x7f0f800b41c0 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f86068640 5 monclient: authenticate success, global_id 16856 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f86068640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f86068640 4 set_mon_vals no callback set 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:58:58.576 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.551+0000 7f0f7effd640 10 monclient: dump: 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 10 monclient: shutdown 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:58.577 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 -- 172.21.15.139:0/2157236686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 msgr2=0x7f0f800b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 --2- 172.21.15.139:0/2157236686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f800b45e0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7f0f680098a0 tx=0x7f0f6802b840 comp rx=0 tx=0).stop 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 -- 172.21.15.139:0/2157236686 shutdown_connections 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 --2- 172.21.15.139:0/2157236686 >> v2:172.21.15.177:3300/0 conn(0x7f0f8014c1d0 0x7f0f8014e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 --2- 172.21.15.139:0/2157236686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f800b45e0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 -- 172.21.15.139:0/2157236686 >> 172.21.15.139:0/2157236686 conn(0x7f0f800b62c0 msgr2=0x7f0f800b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 -- 172.21.15.139:0/2157236686 shutdown_connections 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 1 -- 172.21.15.139:0/2157236686 wait complete. 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 10 monclient: build_initial_monmap 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.552+0000 7f0f86068640 10 monclient: monmap: 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:58.578 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 1 librados: starting msgr at 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 1 librados: starting objecter 2024-07-26T23:58:58.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 1 Processor -- start 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 1 -- start start 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 1 librados: setting wanted keys 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 1 librados: calling monclient init 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 10 monclient: init 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.553+0000 7f0f86068640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f80142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): picked mon.noname-c con 0x7f0f800b41c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f801470c0 0x7f0f8013d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): picked mon.noname-a con 0x7f0f801470c0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f8014c1d0 0x7f0f8013d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8014c1d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80141410 con 0x7f0f801470c0 2024-07-26T23:58:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80074850 con 0x7f0f8014c1d0 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f0f800b4b20 con 0x7f0f800b41c0 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f86068640 10 monclient(hunting): authenticate will time out at 2771.590075s 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f8014c1d0 0x7f0f8013d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f76ffd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f801470c0 0x7f0f8013d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7f7fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f80142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7f7fe640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f80142300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:48514/0 (socket says 172.21.15.139:48514) 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7f7fe640 1 -- 172.21.15.139:0/961642581 learned_addr learned my addr 172.21.15.139:0/961642581 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7f7fe640 10 monclient(hunting): get_auth_request con 0x7f0f800b41c0 auth_method 0 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7ffff640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f8014c1d0 0x7f0f8013d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:46996/0 (socket says 172.21.15.139:46996) 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7ffff640 10 monclient(hunting): get_auth_request con 0x7f0f8014c1d0 auth_method 0 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7ffff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f76ffd640 10 monclient(hunting): get_auth_request con 0x7f0f801470c0 auth_method 0 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f76ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.554+0000 7f0f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.582 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.555+0000 7f0f76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 10 monclient(hunting): handle_auth_done global_id 16859 payload 824 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 1 -- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f8014c1d0 msgr2=0x7f0f8013d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f8014c1d0 0x7f0f8013d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 1 -- 172.21.15.139:0/961642581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f801470c0 msgr2=0x7f0f8013d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0f801470c0 0x7f0f8013d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 10 monclient: _finish_hunting 0 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 1 monclient: found mon.noname-c 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:58:58.583 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.558+0000 7f0f7f7fe640 1 -- 172.21.15.139:0/961642581 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f801441a0 con 0x7f0f800b41c0 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.560+0000 7f0f7f7fe640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f80142300 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c430 tx=0x7f0f7000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.560+0000 7f0f7cff9640 10 client.?.objecter ms_handle_connect 0x7f0f800b41c0 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.560+0000 7f0f7cff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70011790 con 0x7f0f800b41c0 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: dump: 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.585 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: _finish_auth 0 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: _check_auth_tickets 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f70017070 con 0x7f0f800b41c0 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70011bb0 con 0x7f0f800b41c0 2024-07-26T23:58:58.586 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7cff9640 10 monclient: dump: 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 5 monclient: authenticate success, global_id 16859 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 10 monclient: _renew_subs 2024-07-26T23:58:58.587 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f80144480 con 0x7f0f800b41c0 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f7e7fc640 4 set_mon_vals no callback set 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 10 client.16859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 10 monclient: _renew_subs 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f801416d0 con 0x7f0f800b41c0 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.563+0000 7f0f86068640 1 librados: init done 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.564+0000 7f0f86068640 10 client.16859.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.564+0000 7f0f86068640 15 client.16859.objecter _session_command_op_assign -1 1 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.564+0000 7f0f86068640 20 client.16859.objecter _get_session osd=-1 returning homeless 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.564+0000 7f0f86068640 10 client.16859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.564+0000 7f0f86068640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:58:58.588 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.564+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f800b45e0 con 0x7f0f800b41c0 2024-07-26T23:58:58.600 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.585+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f0f7001c460 con 0x7f0f800b41c0 2024-07-26T23:58:58.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.586+0000 7f0f7cff9640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0f600789f0 0x7f0f6007aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.586+0000 7f0f76ffd640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0f600789f0 0x7f0f6007aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.601 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.586+0000 7f0f76ffd640 10 monclient: get_auth_request con 0x7f0f600789f0 auth_method 0 2024-07-26T23:58:58.602 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.587+0000 7f0f7cff9640 10 client.16859.objecter ms_handle_connect 0x7f0f600789f0 2024-07-26T23:58:58.602 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.587+0000 7f0f76ffd640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0f600789f0 0x7f0f6007aeb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0f68036000 tx=0x7f0f6803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f0f70011d50 con 0x7f0f800b41c0 2024-07-26T23:58:58.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 10 client.16859.objecter ms_dispatch 0x7f0f80000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:58.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 3 client.16859.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 3 client.16859.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 10 client.16859.objecter checking command 1 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f0f60080770 0x7f0f60082c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 20 client.16859.objecter _get_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 20 client.16859.objecter put_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 15 client.16859.objecter _session_command_op_remove -1 1 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 20 client.16859.objecter _get_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 20 client.16859.objecter get_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 15 client.16859.objecter _session_command_op_assign 4 1 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 20 client.16859.objecter put_session s=0x7f0f600800a0 osd=4 4 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 10 client.16859.objecter _send_command 1 2024-07-26T23:58:58.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 --> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f7000ee50 con 0x7f0f60080770 2024-07-26T23:58:58.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 20 client.16859.objecter dump_active .. 0 homeless 2024-07-26T23:58:58.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_get_version_reply(handle=1 version=970) ==== 24+0+0 (secure 0 0 0) 0x7f0f7005e130 con 0x7f0f800b41c0 2024-07-26T23:58:58.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.589+0000 7f0f7cff9640 10 monclient: handle_get_version_reply finishing 1 version 970 2024-07-26T23:58:58.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.590+0000 7f0f7ffff640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f0f60080770 0x7f0f60082c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.590+0000 7f0f7ffff640 10 monclient: get_auth_request con 0x7f0f60080770 auth_method 0 2024-07-26T23:58:58.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.593+0000 7f0f7ffff640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f0f60080770 0x7f0f60082c10 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.593+0000 7f0f7cff9640 10 client.16859.objecter ms_handle_connect 0x7f0f60080770 2024-07-26T23:58:58.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.616+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== osd.4 v2:172.21.15.177:6826/4112782318 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f7000ee50 con 0x7f0f60080770 2024-07-26T23:58:58.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.616+0000 7f0f7cff9640 10 client.16859.objecter ms_dispatch 0x7f0f80000b90 command_reply(tid 1: 0 ) 2024-07-26T23:58:58.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.616+0000 7f0f7cff9640 10 client.16859.objecter _finish_command 1 = system:0 2024-07-26T23:58:58.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.616+0000 7f0f7cff9640 20 client.16859.objecter put_session s=0x7f0f600800a0 osd=4 4 2024-07-26T23:58:58.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.616+0000 7f0f7cff9640 15 client.16859.objecter _session_command_op_remove 4 1 2024-07-26T23:58:58.644 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 10 client.16859.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:58:58.644 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 15 client.16859.objecter _session_command_op_assign -1 2 2024-07-26T23:58:58.644 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 20 client.16859.objecter _get_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.644 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 20 client.16859.objecter put_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.645 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 20 client.16859.objecter _get_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.645 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 15 client.16859.objecter _session_command_op_remove -1 2 2024-07-26T23:58:58.645 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 20 client.16859.objecter get_session s=0x7f0f600800a0 osd=4 3 2024-07-26T23:58:58.645 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 15 client.16859.objecter _session_command_op_assign 4 2 2024-07-26T23:58:58.645 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 20 client.16859.objecter put_session s=0x7f0f600800a0 osd=4 4 2024-07-26T23:58:58.645 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 10 client.16859.objecter _send_command 2 2024-07-26T23:58:58.645 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.629+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 --> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0f80143220 con 0x7f0f60080770 2024-07-26T23:58:58.660 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f7cff9640 1 -- 172.21.15.139:0/961642581 <== osd.4 v2:172.21.15.177:6826/4112782318 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0f80143220 con 0x7f0f60080770 2024-07-26T23:58:58.660 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f7cff9640 10 client.16859.objecter ms_dispatch 0x7f0f80000b90 command_reply(tid 2: 0 ) 2024-07-26T23:58:58.660 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f7cff9640 10 client.16859.objecter _finish_command 2 = system:0 2024-07-26T23:58:58.660 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f7cff9640 20 client.16859.objecter put_session s=0x7f0f600800a0 osd=4 4 2024-07-26T23:58:58.660 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f7cff9640 15 client.16859.objecter _session_command_op_remove 4 2 2024-07-26T23:58:58.660 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:58:58.661 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:58:58.661 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 10 librados: watch_flush enter 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 10 librados: watch_flush exit 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 10 client.16859.objecter close_session for osd.4 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f0f60080770 msgr2=0x7f0f60082c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6826/4112782318,v1:172.21.15.177:6827/4112782318] conn(0x7f0f60080770 0x7f0f60082c10 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 20 client.16859.objecter put_session s=0x7f0f600800a0 osd=4 1 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 20 client.16859.objecter shutdown clearing up homeless session... 2024-07-26T23:58:58.662 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 10 client.16859.objecter successfully canceled tick 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0f600789f0 msgr2=0x7f0f6007aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f0f600789f0 0x7f0f6007aeb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0f68036000 tx=0x7f0f6803d040 comp rx=0 tx=0).stop 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 10 monclient: shutdown 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 msgr2=0x7f0f80142300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 --2- 172.21.15.139:0/961642581 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f0f800b41c0 0x7f0f80142300 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c430 tx=0x7f0f7000c8f0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f7ffff640 1 -- 172.21.15.139:0/961642581 reap_dead start 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 shutdown_connections 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 >> 172.21.15.139:0/961642581 conn(0x7f0f800b62c0 msgr2=0x7f0f800b56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 shutdown_connections 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 -- 172.21.15.139:0/961642581 wait complete. 2024-07-26T23:58:58.663 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.645+0000 7f0f86068640 1 librados: shutdown 2024-07-26T23:58:58.671 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-07-26T23:58:58.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.762+0000 7f7a7caf1640 10 monclient: get_monmap_and_config 2024-07-26T23:58:58.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.762+0000 7f7a7caf1640 10 monclient: build_initial_monmap 2024-07-26T23:58:58.777 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.762+0000 7f7a7caf1640 10 monclient: monmap: 2024-07-26T23:58:58.777 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:58.763630+0000 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:58.763630+0000 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:58.778 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 1 Processor -- start 2024-07-26T23:58:58.780 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 1 -- start start 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient: init 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f7a78068740 0x7f7a7813d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient(hunting): picked mon.noname-a con 0x7f7a78068740 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7a7806fa60 0x7f7a7813d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient(hunting): picked mon.noname-c con 0x7f7a7806fa60 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a78143910 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a780af8f0 con 0x7f7a78143910 2024-07-26T23:58:58.781 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f7a78068b20 con 0x7f7a78068740 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f7a780b0230 con 0x7f7a7806fa60 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.763+0000 7f7a7caf1640 10 monclient(hunting): authenticate will time out at 2771.799877s 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a7659c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f7a78068740 0x7f7a7813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a6dd9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7a7806fa60 0x7f7a7813d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a7659c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f7a78068740 0x7f7a7813d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36416/0 (socket says 172.21.15.139:36416) 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a7659c640 1 -- 172.21.15.139:0/3094964168 learned_addr learned my addr 172.21.15.139:0/3094964168 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a7659c640 10 monclient(hunting): get_auth_request con 0x7f7a78068740 auth_method 0 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a7659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.782 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a7659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a6dd9b640 10 monclient(hunting): get_auth_request con 0x7f7a7806fa60 auth_method 0 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.764+0000 7f7a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.765+0000 7f7a7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.765+0000 7f7a7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.765+0000 7f7a7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.765+0000 7f7a6dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.765+0000 7f7a6dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.783 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.765+0000 7f7a6dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 10 monclient(hunting): handle_auth_done global_id 15582 payload 274 2024-07-26T23:58:58.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 1 -- 172.21.15.139:0/3094964168 >> v2:172.21.15.177:3300/0 conn(0x7f7a7806fa60 msgr2=0x7f7a7813d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:58.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 1 --2- 172.21.15.139:0/3094964168 >> v2:172.21.15.177:3300/0 conn(0x7f7a7806fa60 0x7f7a7813d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 1 -- 172.21.15.139:0/3094964168 >> v1:172.21.15.177:6789/0 conn(0x7f7a78143910 legacy=0x7f7a78143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 10 monclient: _finish_hunting 0 2024-07-26T23:58:58.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 1 monclient: found mon.noname-a 2024-07-26T23:58:58.784 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:58.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:58.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.767+0000 7f7a7659c640 1 -- 172.21.15.139:0/3094964168 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a7806d950 con 0x7f7a78068740 2024-07-26T23:58:58.785 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.768+0000 7f7a7659c640 1 --2- 172.21.15.139:0/3094964168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a78068740 0x7f7a7813d450 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f7a78074a00 tx=0x7f7a6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d2fb197d7df92d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.787 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 1 -- 172.21.15.139:0/3094964168 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68035020 con 0x7f7a78068740 2024-07-26T23:58:58.787 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: dump: 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.788 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: _finish_auth 0 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: _check_auth_tickets 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 1 -- 172.21.15.139:0/3094964168 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a68031870 con 0x7f7a78068740 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 1 -- 172.21.15.139:0/3094964168 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68031c40 con 0x7f7a78068740 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a75d9b640 10 monclient: dump: 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.789 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.790 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.790 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.790 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.790 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a7caf1640 5 monclient: authenticate success, global_id 15582 2024-07-26T23:58:58.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a7caf1640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:58.790 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.772+0000 7f7a7caf1640 4 set_mon_vals no callback set 2024-07-26T23:58:58.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 10 monclient: shutdown 2024-07-26T23:58:58.793 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 -- 172.21.15.139:0/3094964168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a78068740 msgr2=0x7f7a7813d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/3094964168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a78068740 0x7f7a7813d450 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f7a78074a00 tx=0x7f7a6802b910 comp rx=0 tx=0).stop 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 -- 172.21.15.139:0/3094964168 shutdown_connections 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/3094964168 >> v2:172.21.15.177:3300/0 conn(0x7f7a7806fa60 0x7f7a7813d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/3094964168 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a78068740 0x7f7a7813d450 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 -- 172.21.15.139:0/3094964168 >> 172.21.15.139:0/3094964168 conn(0x7f7a7806c720 msgr2=0x7f7a7806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 -- 172.21.15.139:0/3094964168 shutdown_connections 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.773+0000 7f7a7caf1640 1 -- 172.21.15.139:0/3094964168 wait complete. 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 10 monclient: build_initial_monmap 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 10 monclient: monmap: 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 1 librados: starting msgr at 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 1 librados: starting objecter 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 1 Processor -- start 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 1 -- start start 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 1 librados: setting wanted keys 2024-07-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 1 librados: calling monclient init 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.774+0000 7f7a7caf1640 10 monclient: init 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a78068740 0x7f7a78145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): picked mon.noname-b con 0x7f7a78068740 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a7806fa60 0x7f7a7813e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): picked mon.noname-a con 0x7f7a7806fa60 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f7a78143910 0x7f7a781462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): picked mon.noname-c con 0x7f7a78143910 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7814e220 con 0x7f7a7806fa60 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78144430 con 0x7f7a78068740 2024-07-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f7a7814e3a0 con 0x7f7a78143910 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a7caf1640 10 monclient(hunting): authenticate will time out at 2771.811382s 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.775+0000 7f7a6dd9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a7806fa60 0x7f7a7813e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f7a78143910 0x7f7a781462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a7659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a78068740 0x7f7a78145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a7806fa60 0x7f7a7813e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36426/0 (socket says 172.21.15.139:36426) 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 1 -- 172.21.15.139:0/1643358507 learned_addr learned my addr 172.21.15.139:0/1643358507 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 10 monclient(hunting): get_auth_request con 0x7f7a7806fa60 auth_method 0 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 10 monclient(hunting): get_auth_request con 0x7f7a78143910 auth_method 0 2024-07-26T23:58:58.797 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a7659c640 10 monclient(hunting): get_auth_request con 0x7f7a78068740 auth_method 0 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a7659c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a7659c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a6dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:58.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.776+0000 7f7a7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:58.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.777+0000 7f7a7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:58.799 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 10 monclient(hunting): handle_auth_done global_id 15585 payload 824 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 1 -- 172.21.15.139:0/1643358507 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f7a78143910 msgr2=0x7f7a781462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f7a78143910 0x7f7a781462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 1 -- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a78068740 msgr2=0x7f7a78145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a78068740 0x7f7a78145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 10 monclient: _finish_hunting 0 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 1 monclient: found mon.noname-a 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.779+0000 7f7a6dd9b640 1 -- 172.21.15.139:0/1643358507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a78147700 con 0x7f7a7806fa60 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.781+0000 7f7a6dd9b640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a7806fa60 0x7f7a7813e410 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000e9b0 tx=0x7f7a6000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.781+0000 7f7a6f7fe640 10 client.?.objecter ms_handle_connect 0x7f7a7806fa60 2024-07-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.781+0000 7f7a6f7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 1 -- 172.21.15.139:0/1643358507 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60017070 con 0x7f7a7806fa60 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: dump: 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.802 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 1 -- 172.21.15.139:0/1643358507 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a6000c9a0 con 0x7f7a7806fa60 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 5 monclient: authenticate success, global_id 15585 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 1 -- 172.21.15.139:0/1643358507 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6000cd90 con 0x7f7a7806fa60 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:58.803 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a6f7fe640 10 monclient: dump: 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 10 monclient: _renew_subs 2024-07-26T23:58:58.804 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a781479e0 con 0x7f7a7806fa60 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7559a640 4 set_mon_vals no callback set 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 10 client.15585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 10 monclient: _renew_subs 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a78074150 con 0x7f7a7806fa60 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.785+0000 7f7a7caf1640 1 librados: init done 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.786+0000 7f7a7caf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.786+0000 7f7a7caf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.786+0000 7f7a7caf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:58:58.805 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.786+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a78074470 con 0x7f7a7806fa60 2024-07-26T23:58:58.821 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.806+0000 7f7a6f7fe640 1 -- 172.21.15.139:0/1643358507 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f7a6001a070 con 0x7f7a7806fa60 2024-07-26T23:58:58.821 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.806+0000 7f7a6f7fe640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f7a580789f0 0x7f7a5807aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:58.822 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.807+0000 7f7a7659c640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f7a580789f0 0x7f7a5807aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:58.822 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.807+0000 7f7a7659c640 10 monclient: get_auth_request con 0x7f7a580789f0 auth_method 0 2024-07-26T23:58:58.822 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.807+0000 7f7a6f7fe640 10 client.15585.objecter ms_handle_connect 0x7f7a580789f0 2024-07-26T23:58:58.822 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.807+0000 7f7a7659c640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f7a580789f0 0x7f7a5807aeb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7a68036000 tx=0x7f7a6803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:58.825 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.810+0000 7f7a6f7fe640 1 -- 172.21.15.139:0/1643358507 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f7a60014db0 con 0x7f7a7806fa60 2024-07-26T23:58:58.825 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.810+0000 7f7a6f7fe640 10 client.15585.objecter ms_dispatch 0x7f7a78000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:58.825 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.810+0000 7f7a6f7fe640 3 client.15585.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.810+0000 7f7a6f7fe640 3 client.15585.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:58.810+0000 7f7a6f7fe640 20 client.15585.objecter dump_active .. 0 homeless 2024-07-26T23:58:59.063 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.048+0000 7f7a6f7fe640 1 -- 172.21.15.139:0/1643358507 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f7a60061ab0 con 0x7f7a7806fa60 2024-07-26T23:58:59.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.048+0000 7f7a6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:59.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.048+0000 7f7a6f7fe640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:58:59.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.244+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f7a78143d20 con 0x7f7a580789f0 2024-07-26T23:58:59.260 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.245+0000 7f7a6f7fe640 1 -- 172.21.15.139:0/1643358507 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7a78143d20 con 0x7f7a580789f0 2024-07-26T23:58:59.260 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 4 to deep-scrub 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 10 librados: watch_flush enter 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 10 librados: watch_flush exit 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 20 client.15585.objecter shutdown clearing up homeless session... 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 10 client.15585.objecter successfully canceled tick 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f7a580789f0 msgr2=0x7f7a5807aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f7a580789f0 0x7f7a5807aeb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7a68036000 tx=0x7f7a6803c040 comp rx=0 tx=0).stop 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 10 monclient: shutdown 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a7806fa60 msgr2=0x7f7a7813e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a7806fa60 0x7f7a7813e410 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000e9b0 tx=0x7f7a6000ee70 comp rx=0 tx=0).stop 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 shutdown_connections 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f7a580789f0 0x7f7a5807aeb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f7a78143910 0x7f7a781462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a7806fa60 0x7f7a7813e410 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 --2- 172.21.15.139:0/1643358507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a78068740 0x7f7a78145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 >> 172.21.15.139:0/1643358507 conn(0x7f7a7806c720 msgr2=0x7f7a7813afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:59.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 shutdown_connections 2024-07-26T23:58:59.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 -- 172.21.15.139:0/1643358507 wait complete. 2024-07-26T23:58:59.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.247+0000 7f7a7caf1640 1 librados: shutdown 2024-07-26T23:58:59.273 INFO:tasks.ceph:Scrubbing osd.5 2024-07-26T23:58:59.273 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:58:59.381 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.366+0000 7f4d69229640 10 monclient: get_monmap_and_config 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.366+0000 7f4d69229640 10 monclient: build_initial_monmap 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.366+0000 7f4d69229640 10 monclient: monmap: 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:59.367704+0000 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:59.367704+0000 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:59.382 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:59.383 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.367+0000 7f4d69229640 1 Processor -- start 2024-07-26T23:58:59.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.367+0000 7f4d69229640 1 -- start start 2024-07-26T23:58:59.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.367+0000 7f4d69229640 10 monclient: init 2024-07-26T23:58:59.388 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.367+0000 7f4d69229640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.367+0000 7f4d69229640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d640b2660 addr v1:172.21.15.177:6789/0 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4d640b2a40 0x7f4d64143c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 10 monclient(hunting): picked mon.noname-c con 0x7f4d640b2a40 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f4d641441a0 0x7f4d6414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 10 monclient(hunting): picked mon.noname-a con 0x7f4d641441a0 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d640af400 con 0x7f4d640b2660 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f4d640af6e0 con 0x7f4d641441a0 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f4d640ad560 con 0x7f4d640b2a40 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:59.389 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d69229640 10 monclient(hunting): authenticate will time out at 2772.404063s 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d6359e640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f4d641441a0 0x7f4d6414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d5bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4d640b2a40 0x7f4d64143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d6359e640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f4d641441a0 0x7f4d6414aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36432/0 (socket says 172.21.15.139:36432) 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d6359e640 1 -- 172.21.15.139:0/1558070681 learned_addr learned my addr 172.21.15.139:0/1558070681 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d6359e640 10 monclient(hunting): get_auth_request con 0x7f4d641441a0 auth_method 0 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d6359e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.368+0000 7f4d6359e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d5bfff640 10 monclient(hunting): get_auth_request con 0x7f4d640b2a40 auth_method 0 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d5bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.390 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.369+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 10 monclient(hunting): handle_auth_done global_id 15588 payload 274 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 1 -- 172.21.15.139:0/1558070681 >> v2:172.21.15.177:3300/0 conn(0x7f4d640b2a40 msgr2=0x7f4d64143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 1 --2- 172.21.15.139:0/1558070681 >> v2:172.21.15.177:3300/0 conn(0x7f4d640b2a40 0x7f4d64143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 1 -- 172.21.15.139:0/1558070681 >> v1:172.21.15.177:6789/0 conn(0x7f4d640b2660 legacy=0x7f4d64074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 10 monclient: _finish_hunting 0 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 1 monclient: found mon.noname-a 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:59.391 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:58:59.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.371+0000 7f4d6359e640 1 -- 172.21.15.139:0/1558070681 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d641447a0 con 0x7f4d641441a0 2024-07-26T23:58:59.392 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.373+0000 7f4d6359e640 1 --2- 172.21.15.139:0/1558070681 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 0x7f4d6414aff0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7f4d640aed70 tx=0x7f4d5c02e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0dce70d6a2d54d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:59.396 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 1 -- 172.21.15.139:0/1558070681 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c02ea00 con 0x7f4d641441a0 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: dump: 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.397 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: _finish_auth 0 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: _check_auth_tickets 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 1 -- 172.21.15.139:0/1558070681 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d5c02eba0 con 0x7f4d641441a0 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 1 -- 172.21.15.139:0/1558070681 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c009600 con 0x7f4d641441a0 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d6259c640 10 monclient: dump: 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.398 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d69229640 5 monclient: authenticate success, global_id 15588 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d69229640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.376+0000 7f4d69229640 4 set_mon_vals no callback set 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 10 monclient: shutdown 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:59.399 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 -- 172.21.15.139:0/1558070681 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 msgr2=0x7f4d6414aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 --2- 172.21.15.139:0/1558070681 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 0x7f4d6414aff0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7f4d640aed70 tx=0x7f4d5c02e060 comp rx=0 tx=0).stop 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 -- 172.21.15.139:0/1558070681 shutdown_connections 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 --2- 172.21.15.139:0/1558070681 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 0x7f4d6414aff0 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 --2- 172.21.15.139:0/1558070681 >> v2:172.21.15.177:3300/0 conn(0x7f4d640b2a40 0x7f4d64143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 -- 172.21.15.139:0/1558070681 >> 172.21.15.139:0/1558070681 conn(0x7f4d640b5b40 msgr2=0x7f4d640b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 -- 172.21.15.139:0/1558070681 shutdown_connections 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.377+0000 7f4d69229640 1 -- 172.21.15.139:0/1558070681 wait complete. 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 10 monclient: build_initial_monmap 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 10 monclient: monmap: 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:59.400 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 1 librados: starting msgr at 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 1 librados: starting objecter 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 1 Processor -- start 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 1 -- start start 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 1 librados: setting wanted keys 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 1 librados: calling monclient init 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.378+0000 7f4d69229640 10 monclient: init 2024-07-26T23:58:59.401 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640b2660 0x7f4d64138f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): picked mon.noname-b con 0x7f4d640b2660 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 0x7f4d641d7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): picked mon.noname-a con 0x7f4d641441a0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4d64139ba0 0x7f4d6413a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): picked mon.noname-c con 0x7f4d64139ba0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4d640ae3a0 con 0x7f4d641441a0 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4d640ae220 con 0x7f4d640b2660 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.402 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f4d64074a80 con 0x7f4d64139ba0 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d69229640 10 monclient(hunting): authenticate will time out at 2772.415366s 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d5bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 0x7f4d641d7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d6359e640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4d64139ba0 0x7f4d6413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.379+0000 7f4d62d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640b2660 0x7f4d64138f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640b2660 0x7f4d64138f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:47042/0 (socket says 172.21.15.139:47042) 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 0x7f4d641d7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36440/0 (socket says 172.21.15.139:36440) 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 1 -- 172.21.15.139:0/1882269294 learned_addr learned my addr 172.21.15.139:0/1882269294 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 10 monclient(hunting): get_auth_request con 0x7f4d640b2660 auth_method 0 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.403 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 10 monclient(hunting): get_auth_request con 0x7f4d641441a0 auth_method 0 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d6359e640 10 monclient(hunting): get_auth_request con 0x7f4d64139ba0 auth_method 0 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d6359e640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d6359e640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.404 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.380+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.381+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.381+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.405 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.381+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 10 monclient(hunting): handle_auth_done global_id 20338 payload 824 2024-07-26T23:58:59.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 1 -- 172.21.15.139:0/1882269294 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4d64139ba0 msgr2=0x7f4d6413a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4d64139ba0 0x7f4d6413a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.407 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 1 -- 172.21.15.139:0/1882269294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 msgr2=0x7f4d641d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4d641441a0 0x7f4d641d7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 10 monclient: _finish_hunting 0 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 1 monclient: found mon.noname-b 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.383+0000 7f4d62d9d640 1 -- 172.21.15.139:0/1882269294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6413a780 con 0x7f4d640b2660 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.385+0000 7f4d62d9d640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640b2660 0x7f4d64138f70 secure :-1 s=READY pgs=4947 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c3d0 tx=0x7f4d5400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.385+0000 7f4d5b7fe640 10 client.?.objecter ms_handle_connect 0x7f4d640b2660 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.385+0000 7f4d5b7fe640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54017070 con 0x7f4d640b2660 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:59.408 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: dump: 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: _finish_auth 0 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:59.409 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d540052d0 con 0x7f4d640b2660 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54003830 con 0x7f4d640b2660 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d5b7fe640 10 monclient: dump: 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.410 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 5 monclient: authenticate success, global_id 20338 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 10 monclient: _renew_subs 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6413d180 con 0x7f4d640b2660 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d61d9b640 4 set_mon_vals no callback set 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 10 client.20338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 10 monclient: _renew_subs 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d64137950 con 0x7f4d640b2660 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.389+0000 7f4d69229640 1 librados: init done 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.390+0000 7f4d69229640 10 client.20338.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:59.411 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.390+0000 7f4d69229640 15 client.20338.objecter _session_command_op_assign -1 1 2024-07-26T23:58:59.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.390+0000 7f4d69229640 20 client.20338.objecter _get_session osd=-1 returning homeless 2024-07-26T23:58:59.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.390+0000 7f4d69229640 10 client.20338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:59.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.390+0000 7f4d69229640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.412 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.390+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d64143c60 con 0x7f4d640b2660 2024-07-26T23:58:59.425 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.410+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f4d5401a070 con 0x7f4d640b2660 2024-07-26T23:58:59.425 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.411+0000 7f4d5b7fe640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f4d44078a20 0x7f4d4407aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.411+0000 7f4d5bfff640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f4d44078a20 0x7f4d4407aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.411+0000 7f4d5bfff640 10 monclient: get_auth_request con 0x7f4d44078a20 auth_method 0 2024-07-26T23:58:59.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.411+0000 7f4d5bfff640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f4d44078a20 0x7f4d4407aee0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d641394b0 tx=0x7f4d4c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:59.426 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.411+0000 7f4d5b7fe640 10 client.20338.objecter ms_handle_connect 0x7f4d44078a20 2024-07-26T23:58:59.430 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.414+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f4d5405cfe0 con 0x7f4d640b2660 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.414+0000 7f4d5b7fe640 10 client.20338.objecter ms_dispatch 0x7f4d64000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.414+0000 7f4d5b7fe640 3 client.20338.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.414+0000 7f4d5b7fe640 3 client.20338.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 10 client.20338.objecter checking command 1 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7f4d44080800 0x7f4d44082ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 20 client.20338.objecter _get_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 20 client.20338.objecter put_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 15 client.20338.objecter _session_command_op_remove -1 1 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 20 client.20338.objecter _get_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 20 client.20338.objecter get_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 15 client.20338.objecter _session_command_op_assign 5 1 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 20 client.20338.objecter put_session s=0x7f4d44080130 osd=5 4 2024-07-26T23:58:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 10 client.20338.objecter _send_command 1 2024-07-26T23:58:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 --> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d5400ae50 con 0x7f4d44080800 2024-07-26T23:58:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 20 client.20338.objecter dump_active .. 0 homeless 2024-07-26T23:58:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=970) ==== 24+0+0 (secure 0 0 0) 0x7f4d5405d3a0 con 0x7f4d640b2660 2024-07-26T23:58:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d5b7fe640 10 monclient: handle_get_version_reply finishing 1 version 970 2024-07-26T23:58:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d6359e640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7f4d44080800 0x7f4d44082ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.415+0000 7f4d6359e640 10 monclient: get_auth_request con 0x7f4d44080800 auth_method 0 2024-07-26T23:58:59.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.418+0000 7f4d6359e640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7f4d44080800 0x7f4d44082ca0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:59.433 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.418+0000 7f4d5b7fe640 10 client.20338.objecter ms_handle_connect 0x7f4d44080800 2024-07-26T23:58:59.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.441+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== osd.5 v2:172.21.15.177:6802/228413488 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d5400ae50 con 0x7f4d44080800 2024-07-26T23:58:59.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.441+0000 7f4d5b7fe640 10 client.20338.objecter ms_dispatch 0x7f4d64000b90 command_reply(tid 1: 0 ) 2024-07-26T23:58:59.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.441+0000 7f4d5b7fe640 10 client.20338.objecter _finish_command 1 = system:0 2024-07-26T23:58:59.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.441+0000 7f4d5b7fe640 20 client.20338.objecter put_session s=0x7f4d44080130 osd=5 4 2024-07-26T23:58:59.457 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.441+0000 7f4d5b7fe640 15 client.20338.objecter _session_command_op_remove 5 1 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 10 client.20338.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 15 client.20338.objecter _session_command_op_assign -1 2 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 20 client.20338.objecter _get_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 20 client.20338.objecter put_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 20 client.20338.objecter _get_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 15 client.20338.objecter _session_command_op_remove -1 2 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 20 client.20338.objecter get_session s=0x7f4d44080130 osd=5 3 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 15 client.20338.objecter _session_command_op_assign 5 2 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 20 client.20338.objecter put_session s=0x7f4d44080130 osd=5 4 2024-07-26T23:58:59.470 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 10 client.20338.objecter _send_command 2 2024-07-26T23:58:59.471 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.454+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 --> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4d6413ef90 con 0x7f4d44080800 2024-07-26T23:58:59.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.468+0000 7f4d5b7fe640 1 -- 172.21.15.139:0/1882269294 <== osd.5 v2:172.21.15.177:6802/228413488 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4d6413ef90 con 0x7f4d44080800 2024-07-26T23:58:59.483 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.468+0000 7f4d5b7fe640 10 client.20338.objecter ms_dispatch 0x7f4d64000b90 command_reply(tid 2: 0 ) 2024-07-26T23:58:59.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.468+0000 7f4d5b7fe640 10 client.20338.objecter _finish_command 2 = system:0 2024-07-26T23:58:59.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.468+0000 7f4d5b7fe640 20 client.20338.objecter put_session s=0x7f4d44080130 osd=5 4 2024-07-26T23:58:59.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.468+0000 7f4d5b7fe640 15 client.20338.objecter _session_command_op_remove 5 2 2024-07-26T23:58:59.484 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:58:59.484 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:58:59.484 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 10 librados: watch_flush enter 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 10 librados: watch_flush exit 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 10 client.20338.objecter close_session for osd.5 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7f4d44080800 msgr2=0x7f4d44082ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6802/228413488,v1:172.21.15.177:6803/228413488] conn(0x7f4d44080800 0x7f4d44082ca0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 20 client.20338.objecter put_session s=0x7f4d44080130 osd=5 1 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 20 client.20338.objecter shutdown clearing up homeless session... 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 10 client.20338.objecter successfully canceled tick 2024-07-26T23:58:59.486 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f4d44078a20 msgr2=0x7f4d4407aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f4d44078a20 0x7f4d4407aee0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d641394b0 tx=0x7f4d4c009190 comp rx=0 tx=0).stop 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 10 monclient: shutdown 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640b2660 msgr2=0x7f4d64138f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 --2- 172.21.15.139:0/1882269294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640b2660 0x7f4d64138f70 secure :-1 s=READY pgs=4947 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c3d0 tx=0x7f4d5400c890 comp rx=0 tx=0).stop 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d6359e640 1 -- 172.21.15.139:0/1882269294 reap_dead start 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 shutdown_connections 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 >> 172.21.15.139:0/1882269294 conn(0x7f4d640b5b40 msgr2=0x7f4d6413f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 shutdown_connections 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 -- 172.21.15.139:0/1882269294 wait complete. 2024-07-26T23:58:59.487 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.469+0000 7f4d69229640 1 librados: shutdown 2024-07-26T23:58:59.495 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-07-26T23:58:59.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.582+0000 7f8e54ccd640 10 monclient: get_monmap_and_config 2024-07-26T23:58:59.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.582+0000 7f8e54ccd640 10 monclient: build_initial_monmap 2024-07-26T23:58:59.598 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.582+0000 7f8e54ccd640 10 monclient: monmap: 2024-07-26T23:58:59.598 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:59.599 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:58:59.584015+0000 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:58:59.584015+0000 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:58:59.600 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.603 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.583+0000 7f8e54ccd640 1 Processor -- start 2024-07-26T23:58:59.603 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.583+0000 7f8e54ccd640 1 -- start start 2024-07-26T23:58:59.603 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.583+0000 7f8e54ccd640 10 monclient: init 2024-07-26T23:58:59.603 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e50072be0 addr v1:172.21.15.139:6789/0 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8e50072fc0 0x7f8e50141660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient(hunting): picked mon.noname-c con 0x7f8e50072fc0 addr v2:172.21.15.177:3300/0 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8e50071ad0 0x7f8e50141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient(hunting): picked mon.noname-a con 0x7f8e50071ad0 addr v2:172.21.15.139:3300/0 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e50074350 con 0x7f8e50072be0 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f8e50072510 con 0x7f8e50071ad0 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8e500b3d30 con 0x7f8e50072fc0 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:59.604 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.584+0000 7f8e54ccd640 10 monclient(hunting): authenticate will time out at 2772.620542s 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4ed9d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8e50071ad0 0x7f8e50141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4dd9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8e50072fc0 0x7f8e50141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4ed9d640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8e50071ad0 0x7f8e50141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:36452/0 (socket says 172.21.15.139:36452) 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4dd9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8e50072fc0 0x7f8e50141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:47050/0 (socket says 172.21.15.139:47050) 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4ed9d640 1 -- 172.21.15.139:0/3333250713 learned_addr learned my addr 172.21.15.139:0/3333250713 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4ed9d640 10 monclient(hunting): get_auth_request con 0x7f8e50071ad0 auth_method 0 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4dd9b640 10 monclient(hunting): get_auth_request con 0x7f8e50072fc0 auth_method 0 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.605 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.585+0000 7f8e4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.586+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.586+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.586+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.586+0000 7f8e4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.586+0000 7f8e4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.606 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.586+0000 7f8e4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_done global_id 20341 payload 274 2024-07-26T23:58:59.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 1 -- 172.21.15.139:0/3333250713 >> v2:172.21.15.139:3300/0 conn(0x7f8e50071ad0 msgr2=0x7f8e50141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:58:59.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 1 --2- 172.21.15.139:0/3333250713 >> v2:172.21.15.139:3300/0 conn(0x7f8e50071ad0 0x7f8e50141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 1 -- 172.21.15.139:0/3333250713 >> v1:172.21.15.139:6789/0 conn(0x7f8e50072be0 legacy=0x7f8e500b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.607 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 10 monclient: _finish_hunting 0 2024-07-26T23:58:59.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 1 monclient: found mon.noname-c 2024-07-26T23:58:59.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:59.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.608 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.587+0000 7f8e4dd9b640 1 -- 172.21.15.139:0/3333250713 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e501420e0 con 0x7f8e50072fc0 2024-07-26T23:58:59.615 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.589+0000 7f8e4dd9b640 1 --2- 172.21.15.139:0/3333250713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072fc0 0x7f8e50141660 secure :-1 s=READY pgs=4948 cs=0 l=1 rev1=1 crypto rx=0x7f8e5013f400 tx=0x7f8e4402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72e86dcb27605bca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 1 -- 172.21.15.139:0/3333250713 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e440315a0 con 0x7f8e50072fc0 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: dump: 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.616 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: _finish_auth 0 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: _check_auth_tickets 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 1 -- 172.21.15.139:0/3333250713 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e44037070 con 0x7f8e50072fc0 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 1 -- 172.21.15.139:0/3333250713 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44031960 con 0x7f8e50072fc0 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e54ccd640 5 monclient: authenticate success, global_id 20341 2024-07-26T23:58:59.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e54ccd640 10 monclient: get_monmap_and_config success 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e54ccd640 4 set_mon_vals no callback set 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e4d59a640 10 monclient: dump: 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.618 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e54ccd640 10 monclient: shutdown 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e54ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3333250713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072fc0 msgr2=0x7f8e50141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.593+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3333250713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072fc0 0x7f8e50141660 secure :-1 s=READY pgs=4948 cs=0 l=1 rev1=1 crypto rx=0x7f8e5013f400 tx=0x7f8e4402b840 comp rx=0 tx=0).stop 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3333250713 shutdown_connections 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3333250713 >> v2:172.21.15.139:3300/0 conn(0x7f8e50071ad0 0x7f8e50141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3333250713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072fc0 0x7f8e50141660 unknown :-1 s=CLOSED pgs=4948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3333250713 >> 172.21.15.139:0/3333250713 conn(0x7f8e500b7ac0 msgr2=0x7f8e500b7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3333250713 shutdown_connections 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3333250713 wait complete. 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 10 monclient: build_initial_monmap 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.594+0000 7f8e54ccd640 10 monclient: monmap: 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:58:59.620 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 librados: starting msgr at 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 librados: starting objecter 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 Processor -- start 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 -- start start 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 librados: setting wanted keys 2024-07-26T23:58:59.621 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 librados: calling monclient init 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 10 monclient: init 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 10 monclient: _reopen_session rank -1 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 10 monclient: _add_conns ranks=[2,1,0] 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e50071ad0 0x7f8e5014a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 10 monclient(hunting): picked mon.noname-c con 0x7f8e50071ad0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072be0 0x7f8e5006ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 10 monclient(hunting): picked mon.noname-b con 0x7f8e50072be0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e5006d830 0x7f8e5006dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 10 monclient(hunting): picked mon.noname-a con 0x7f8e5006d830 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.595+0000 7f8e54ccd640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500b1d90 con 0x7f8e5006d830 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e54ccd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e54ccd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500b1c10 con 0x7f8e50072be0 2024-07-26T23:58:59.622 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e54ccd640 10 monclient(hunting): start opening mon connection 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e54ccd640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f8e5014bf70 con 0x7f8e50071ad0 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e54ccd640 10 monclient(hunting): _renew_subs 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e54ccd640 10 monclient(hunting): authenticate will time out at 2772.632018s 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4e59c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e50071ad0 0x7f8e5014a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072be0 0x7f8e5006ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4ed9d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e5006d830 0x7f8e5006dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072be0 0x7f8e5006ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:47062/0 (socket says 172.21.15.139:47062) 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4dd9b640 1 -- 172.21.15.139:0/3621459021 learned_addr learned my addr 172.21.15.139:0/3621459021 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4dd9b640 10 monclient(hunting): get_auth_request con 0x7f8e50072be0 auth_method 0 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4dd9b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.623 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4ed9d640 10 monclient(hunting): get_auth_request con 0x7f8e5006d830 auth_method 0 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4ed9d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4e59c640 10 monclient(hunting): get_auth_request con 0x7f8e50071ad0 auth_method 0 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4e59c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.596+0000 7f8e4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.624 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.597+0000 7f8e4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 10 monclient(hunting): handle_auth_done global_id 20344 payload 824 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 1 -- 172.21.15.139:0/3621459021 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e50071ad0 msgr2=0x7f8e5014a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e50071ad0 0x7f8e5014a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 1 -- 172.21.15.139:0/3621459021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e5006d830 msgr2=0x7f8e5006dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e5006d830 0x7f8e5006dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:58:59.625 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 10 monclient: _finish_hunting 0 2024-07-26T23:58:59.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 1 monclient: found mon.noname-b 2024-07-26T23:58:59.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:58:59.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.626 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.600+0000 7f8e4dd9b640 1 -- 172.21.15.139:0/3621459021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5013c6c0 con 0x7f8e50072be0 2024-07-26T23:58:59.627 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.602+0000 7f8e4dd9b640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072be0 0x7f8e5006ea00 secure :-1 s=READY pgs=4949 cs=0 l=1 rev1=1 crypto rx=0x7f8e4402be50 tx=0x7f8e440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:59.627 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.602+0000 7f8e3effd640 10 client.?.objecter ms_handle_connect 0x7f8e50072be0 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.602+0000 7f8e3effd640 10 client.?.objecter resend_mon_ops 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 1 -- 172.21.15.139:0/3621459021 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44032630 con 0x7f8e50072be0 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: dump: 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.628 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: _finish_auth 0 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: _check_auth_tickets 2024-07-26T23:58:59.629 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 1 -- 172.21.15.139:0/3621459021 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e440327d0 con 0x7f8e50072be0 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 1 -- 172.21.15.139:0/3621459021 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44032bc0 con 0x7f8e50072be0 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e3effd640 10 monclient: dump: 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:58:59.630 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e54ccd640 5 monclient: authenticate success, global_id 20344 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e54ccd640 10 monclient: _renew_subs 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e54ccd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e4cd99640 4 set_mon_vals no callback set 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.605+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5006f800 con 0x7f8e50072be0 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 10 client.20344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:58:59.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 10 monclient: _renew_subs 2024-07-26T23:58:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5006efb0 con 0x7f8e50072be0 2024-07-26T23:58:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 1 librados: init done 2024-07-26T23:58:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:58:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.606+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e500b3620 con 0x7f8e50072be0 2024-07-26T23:58:59.642 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.627+0000 7f8e3effd640 1 -- 172.21.15.139:0/3621459021 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f8e4404c020 con 0x7f8e50072be0 2024-07-26T23:58:59.642 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.627+0000 7f8e3effd640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e30078a20 0x7f8e3007aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:58:59.643 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.627+0000 7f8e4e59c640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e30078a20 0x7f8e3007aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:58:59.643 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.627+0000 7f8e4e59c640 10 monclient: get_auth_request con 0x7f8e30078a20 auth_method 0 2024-07-26T23:58:59.643 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.628+0000 7f8e4e59c640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e30078a20 0x7f8e3007aee0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8e501386e0 tx=0x7f8e40009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:58:59.643 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.628+0000 7f8e3effd640 10 client.20344.objecter ms_handle_connect 0x7f8e30078a20 2024-07-26T23:58:59.869 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.853+0000 7f8e3effd640 1 -- 172.21.15.139:0/3621459021 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f8e4402f020 con 0x7f8e50072be0 2024-07-26T23:58:59.869 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.853+0000 7f8e3effd640 10 client.20344.objecter ms_dispatch 0x7f8e50000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:58:59.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.853+0000 7f8e3effd640 3 client.20344.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:58:59.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.854+0000 7f8e3effd640 3 client.20344.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:58:59.870 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.854+0000 7f8e3effd640 20 client.20344.objecter dump_active .. 0 homeless 2024-07-26T23:58:59.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.865+0000 7f8e3effd640 1 -- 172.21.15.139:0/3621459021 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f8e44040070 con 0x7f8e50072be0 2024-07-26T23:58:59.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.865+0000 7f8e3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:58:59.880 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:58:59.865+0000 7f8e3effd640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:59:00.086 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.071+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f8e5006f650 con 0x7f8e30078a20 2024-07-26T23:59:00.087 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.071+0000 7f8e3effd640 1 -- 172.21.15.139:0/3621459021 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8e5006f650 con 0x7f8e30078a20 2024-07-26T23:59:00.087 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 5 to deep-scrub 2024-07-26T23:59:00.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 10 librados: watch_flush enter 2024-07-26T23:59:00.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 10 librados: watch_flush exit 2024-07-26T23:59:00.090 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 20 client.20344.objecter shutdown clearing up homeless session... 2024-07-26T23:59:00.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 10 client.20344.objecter successfully canceled tick 2024-07-26T23:59:00.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e30078a20 msgr2=0x7f8e3007aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e30078a20 0x7f8e3007aee0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8e501386e0 tx=0x7f8e40009040 comp rx=0 tx=0).stop 2024-07-26T23:59:00.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 10 monclient: shutdown 2024-07-26T23:59:00.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:00.091 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072be0 msgr2=0x7f8e5006ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072be0 0x7f8e5006ea00 secure :-1 s=READY pgs=4949 cs=0 l=1 rev1=1 crypto rx=0x7f8e4402be50 tx=0x7f8e440046c0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 shutdown_connections 2024-07-26T23:59:00.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8e30078a20 0x7f8e3007aee0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8e5006d830 0x7f8e5006dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.092 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e50072be0 0x7f8e5006ea00 unknown :-1 s=CLOSED pgs=4949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 --2- 172.21.15.139:0/3621459021 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8e50071ad0 0x7f8e5014a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 >> 172.21.15.139:0/3621459021 conn(0x7f8e500b7ac0 msgr2=0x7f8e500b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:00.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 shutdown_connections 2024-07-26T23:59:00.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 -- 172.21.15.139:0/3621459021 wait complete. 2024-07-26T23:59:00.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.073+0000 7f8e54ccd640 1 librados: shutdown 2024-07-26T23:59:00.102 INFO:tasks.ceph:Scrubbing osd.6 2024-07-26T23:59:00.103 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:59:00.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.192+0000 7f8f19fdf640 10 monclient: get_monmap_and_config 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.192+0000 7f8f19fdf640 10 monclient: build_initial_monmap 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.192+0000 7f8f19fdf640 10 monclient: monmap: 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:59:00.193816+0000 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:59:00.193816+0000 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:59:00.208 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:59:00.209 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:59:00.209 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:59:00.209 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:59:00.209 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.211 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.193+0000 7f8f19fdf640 1 Processor -- start 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.193+0000 7f8f19fdf640 1 -- start start 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.193+0000 7f8f19fdf640 10 monclient: init 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8f140b20e0 0x7f8f14140b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient(hunting): picked mon.noname-a con 0x7f8f140b20e0 addr v2:172.21.15.139:3300/0 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8f141410b0 0x7f8f14147040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient(hunting): picked mon.noname-c con 0x7f8f141410b0 addr v2:172.21.15.177:3300/0 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f14147580 addr v1:172.21.15.139:6789/0 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f140709b0 con 0x7f8f14147580 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7f8f140b24c0 con 0x7f8f140b20e0 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.212 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8f14141500 con 0x7f8f141410b0 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f19fdf640 10 monclient(hunting): authenticate will time out at 2773.230230s 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f137fe640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7f8f140b20e0 0x7f8f14140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.194+0000 7f8f12ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8f141410b0 0x7f8f14147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8f141410b0 0x7f8f14147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:47582/0 (socket says 172.21.15.139:47582) 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 1 -- 172.21.15.139:0/49678743 learned_addr learned my addr 172.21.15.139:0/49678743 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 10 monclient(hunting): get_auth_request con 0x7f8f141410b0 auth_method 0 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f137fe640 10 monclient(hunting): get_auth_request con 0x7f8f140b20e0 auth_method 0 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f137fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.213 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.195+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.196+0000 7f8f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.196+0000 7f8f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.214 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.196+0000 7f8f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_done global_id 20347 payload 274 2024-07-26T23:59:00.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 1 -- 172.21.15.139:0/49678743 >> v2:172.21.15.139:3300/0 conn(0x7f8f140b20e0 msgr2=0x7f8f14140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:00.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 1 --2- 172.21.15.139:0/49678743 >> v2:172.21.15.139:3300/0 conn(0x7f8f140b20e0 0x7f8f14140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 1 -- 172.21.15.139:0/49678743 >> v1:172.21.15.139:6789/0 conn(0x7f8f14147580 legacy=0x7f8f1406c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.215 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:59:00.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 1 monclient: found mon.noname-c 2024-07-26T23:59:00.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:00.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:59:00.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.197+0000 7f8f12ffd640 1 -- 172.21.15.139:0/49678743 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f14070c20 con 0x7f8f141410b0 2024-07-26T23:59:00.216 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.199+0000 7f8f12ffd640 1 --2- 172.21.15.139:0/49678743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141410b0 0x7f8f14147040 secure :-1 s=READY pgs=4950 cs=0 l=1 rev1=1 crypto rx=0x7f8f140b0900 tx=0x7f8f0802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c358e4a274b49757 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:00.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 1 -- 172.21.15.139:0/49678743 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08035020 con 0x7f8f141410b0 2024-07-26T23:59:00.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:59:00.218 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: dump: 2024-07-26T23:59:00.218 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: _finish_auth 0 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: _check_auth_tickets 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:00.219 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 1 -- 172.21.15.139:0/49678743 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f08031870 con 0x7f8f141410b0 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 1 -- 172.21.15.139:0/49678743 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08031c40 con 0x7f8f141410b0 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f19fdf640 5 monclient: authenticate success, global_id 20347 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f19fdf640 10 monclient: get_monmap_and_config success 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f19fdf640 4 set_mon_vals no callback set 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.202+0000 7f8f127fc640 10 monclient: dump: 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.220 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.221 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.221 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.221 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.221 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.221 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.221 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.221 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 10 monclient: shutdown 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 -- 172.21.15.139:0/49678743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141410b0 msgr2=0x7f8f14147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 --2- 172.21.15.139:0/49678743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141410b0 0x7f8f14147040 secure :-1 s=READY pgs=4950 cs=0 l=1 rev1=1 crypto rx=0x7f8f140b0900 tx=0x7f8f0802b910 comp rx=0 tx=0).stop 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 -- 172.21.15.139:0/49678743 shutdown_connections 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 --2- 172.21.15.139:0/49678743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141410b0 0x7f8f14147040 unknown :-1 s=CLOSED pgs=4950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 --2- 172.21.15.139:0/49678743 >> v2:172.21.15.139:3300/0 conn(0x7f8f140b20e0 0x7f8f14140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 -- 172.21.15.139:0/49678743 >> 172.21.15.139:0/49678743 conn(0x7f8f140b2f60 msgr2=0x7f8f140b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 -- 172.21.15.139:0/49678743 shutdown_connections 2024-07-26T23:59:00.225 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.203+0000 7f8f19fdf640 1 -- 172.21.15.139:0/49678743 wait complete. 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.204+0000 7f8f19fdf640 10 monclient: build_initial_monmap 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.204+0000 7f8f19fdf640 10 monclient: monmap: 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.204+0000 7f8f19fdf640 1 librados: starting msgr at 2024-07-26T23:59:00.226 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.204+0000 7f8f19fdf640 1 librados: starting objecter 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.204+0000 7f8f19fdf640 1 Processor -- start 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 -- start start 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 librados: setting wanted keys 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 librados: calling monclient init 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient: init 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f140b20e0 0x7f8f141dc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): picked mon.noname-b con 0x7f8f140b20e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:59:00.227 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8f141410b0 0x7f8f1411ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): picked mon.noname-c con 0x7f8f141410b0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f14147580 0x7f8f1411f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): picked mon.noname-a con 0x7f8f14147580 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f8f140b03c0 con 0x7f8f14147580 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8f140afdd0 con 0x7f8f140b20e0 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f8f140b0540 con 0x7f8f141410b0 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.205+0000 7f8f19fdf640 10 monclient(hunting): authenticate will time out at 2773.241669s 2024-07-26T23:59:00.228 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f12ffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8f141410b0 0x7f8f1411ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f14147580 0x7f8f1411f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f12ffd640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8f141410b0 0x7f8f1411ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:33682/0 (socket says 172.21.15.139:33682) 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f12ffd640 1 -- 172.21.15.139:0/1871132579 learned_addr learned my addr 172.21.15.139:0/1871132579 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f12ffd640 10 monclient(hunting): get_auth_request con 0x7f8f141410b0 auth_method 0 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f12ffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f13fff640 10 monclient(hunting): get_auth_request con 0x7f8f14147580 auth_method 0 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f13fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.206+0000 7f8f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.229 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.207+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.207+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.207+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.207+0000 7f8f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.207+0000 7f8f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.207+0000 7f8f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.207+0000 7f8f137fe640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f140b20e0 0x7f8f141dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.208+0000 7f8f137fe640 10 monclient(hunting): get_auth_request con 0x7f8f140b20e0 auth_method 0 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.208+0000 7f8f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.208+0000 7f8f137fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.208+0000 7f8f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.230 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.208+0000 7f8f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.208+0000 7f8f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.208+0000 7f8f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 10 monclient(hunting): handle_auth_done global_id 16871 payload 824 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 1 -- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f140b20e0 msgr2=0x7f8f141dc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f140b20e0 0x7f8f141dc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 1 -- 172.21.15.139:0/1871132579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f14147580 msgr2=0x7f8f1411f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8f14147580 0x7f8f1411f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 10 monclient: _finish_hunting 0 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 1 monclient: found mon.noname-c 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:59:00.231 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.209+0000 7f8f12ffd640 1 -- 172.21.15.139:0/1871132579 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f141207c0 con 0x7f8f141410b0 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.212+0000 7f8f12ffd640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8f141410b0 0x7f8f1411ead0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f8f0802b950 tx=0x7f8f080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.212+0000 7f8efbfff640 10 client.?.objecter ms_handle_connect 0x7f8f141410b0 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.212+0000 7f8efbfff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08040070 con 0x7f8f141410b0 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 10 monclient: dump: 2024-07-26T23:59:00.233 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 10 monclient: _finish_auth 0 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 10 monclient: _check_auth_tickets 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:00.234 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f08030d00 con 0x7f8f141410b0 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8f19fdf640 5 monclient: authenticate success, global_id 16871 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8f19fdf640 10 monclient: _renew_subs 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8f19fdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.216+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f14120a30 con 0x7f8f141410b0 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8efbfff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08035040 con 0x7f8f141410b0 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8efbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8efbfff640 10 monclient: dump: 2024-07-26T23:59:00.235 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.236 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8f11ffb640 4 set_mon_vals no callback set 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8f19fdf640 10 client.16871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8f19fdf640 10 monclient: _renew_subs 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8f19fdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f141dccb0 con 0x7f8f141410b0 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.217+0000 7f8f19fdf640 1 librados: init done 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.218+0000 7f8f19fdf640 10 client.16871.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:00.237 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.218+0000 7f8f19fdf640 15 client.16871.objecter _session_command_op_assign -1 1 2024-07-26T23:59:00.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.218+0000 7f8f19fdf640 20 client.16871.objecter _get_session osd=-1 returning homeless 2024-07-26T23:59:00.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.218+0000 7f8f19fdf640 10 client.16871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:00.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.218+0000 7f8f19fdf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:59:00.238 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.218+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f14147a80 con 0x7f8f141410b0 2024-07-26T23:59:00.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.241+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f8f0804b060 con 0x7f8f141410b0 2024-07-26T23:59:00.257 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.241+0000 7f8efbfff640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8f00078a20 0x7f8f0007aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.257 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.242+0000 7f8f137fe640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8f00078a20 0x7f8f0007aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.257 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.242+0000 7f8f137fe640 10 monclient: get_auth_request con 0x7f8f00078a20 auth_method 0 2024-07-26T23:59:00.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.242+0000 7f8efbfff640 10 client.16871.objecter ms_handle_connect 0x7f8f00078a20 2024-07-26T23:59:00.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.242+0000 7f8f137fe640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8f00078a20 0x7f8f0007aee0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8f040113e0 tx=0x7f8f0400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:00.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7f8f08045d40 con 0x7f8f141410b0 2024-07-26T23:59:00.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 10 client.16871.objecter ms_dispatch 0x7f8f14000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:59:00.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 3 client.16871.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:59:00.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 3 client.16871.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:59:00.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 10 client.16871.objecter checking command 1 2024-07-26T23:59:00.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7f8f000807a0 0x7f8f00082c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.261 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 20 client.16871.objecter _get_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 20 client.16871.objecter put_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 15 client.16871.objecter _session_command_op_remove -1 1 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 20 client.16871.objecter _get_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 20 client.16871.objecter get_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 15 client.16871.objecter _session_command_op_assign 6 1 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 20 client.16871.objecter put_session s=0x7f8f000800d0 osd=6 4 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 10 client.16871.objecter _send_command 1 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 --> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f08048e80 con 0x7f8f000807a0 2024-07-26T23:59:00.262 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.245+0000 7f8efbfff640 20 client.16871.objecter dump_active .. 0 homeless 2024-07-26T23:59:00.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.246+0000 7f8f13fff640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7f8f000807a0 0x7f8f00082c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.246+0000 7f8f13fff640 10 monclient: get_auth_request con 0x7f8f000807a0 auth_method 0 2024-07-26T23:59:00.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.247+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== mon.2 v2:172.21.15.139:3301/0 6 ==== mon_get_version_reply(handle=1 version=970) ==== 24+0+0 (secure 0 0 0) 0x7f8f0807fb80 con 0x7f8f141410b0 2024-07-26T23:59:00.263 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.247+0000 7f8efbfff640 10 monclient: handle_get_version_reply finishing 1 version 970 2024-07-26T23:59:00.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.249+0000 7f8f13fff640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7f8f000807a0 0x7f8f00082c40 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:00.264 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.249+0000 7f8efbfff640 10 client.16871.objecter ms_handle_connect 0x7f8f000807a0 2024-07-26T23:59:00.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.274+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== osd.6 v2:172.21.15.177:6810/1761005298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8f08048e80 con 0x7f8f000807a0 2024-07-26T23:59:00.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.274+0000 7f8efbfff640 10 client.16871.objecter ms_dispatch 0x7f8f14000b90 command_reply(tid 1: 0 ) 2024-07-26T23:59:00.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.274+0000 7f8efbfff640 10 client.16871.objecter _finish_command 1 = system:0 2024-07-26T23:59:00.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.274+0000 7f8efbfff640 20 client.16871.objecter put_session s=0x7f8f000800d0 osd=6 4 2024-07-26T23:59:00.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.274+0000 7f8efbfff640 15 client.16871.objecter _session_command_op_remove 6 1 2024-07-26T23:59:00.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 10 client.16871.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:59:00.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 15 client.16871.objecter _session_command_op_assign -1 2 2024-07-26T23:59:00.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 20 client.16871.objecter _get_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 20 client.16871.objecter put_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 20 client.16871.objecter _get_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 15 client.16871.objecter _session_command_op_remove -1 2 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 20 client.16871.objecter get_session s=0x7f8f000800d0 osd=6 3 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 15 client.16871.objecter _session_command_op_assign 6 2 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 20 client.16871.objecter put_session s=0x7f8f000800d0 osd=6 4 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 10 client.16871.objecter _send_command 2 2024-07-26T23:59:00.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.287+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 --> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8f1411f550 con 0x7f8f000807a0 2024-07-26T23:59:00.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8efbfff640 1 -- 172.21.15.139:0/1871132579 <== osd.6 v2:172.21.15.177:6810/1761005298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8f1411f550 con 0x7f8f000807a0 2024-07-26T23:59:00.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8efbfff640 10 client.16871.objecter ms_dispatch 0x7f8f14000b90 command_reply(tid 2: 0 ) 2024-07-26T23:59:00.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8efbfff640 10 client.16871.objecter _finish_command 2 = system:0 2024-07-26T23:59:00.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8efbfff640 20 client.16871.objecter put_session s=0x7f8f000800d0 osd=6 4 2024-07-26T23:59:00.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8efbfff640 15 client.16871.objecter _session_command_op_remove 6 2 2024-07-26T23:59:00.316 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:59:00.316 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:59:00.316 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:59:00.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 10 librados: watch_flush enter 2024-07-26T23:59:00.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 10 librados: watch_flush exit 2024-07-26T23:59:00.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 10 client.16871.objecter close_session for osd.6 2024-07-26T23:59:00.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7f8f000807a0 msgr2=0x7f8f00082c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.318 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6810/1761005298,v1:172.21.15.177:6811/1761005298] conn(0x7f8f000807a0 0x7f8f00082c40 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 20 client.16871.objecter put_session s=0x7f8f000800d0 osd=6 1 2024-07-26T23:59:00.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 20 client.16871.objecter shutdown clearing up homeless session... 2024-07-26T23:59:00.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 10 client.16871.objecter successfully canceled tick 2024-07-26T23:59:00.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8f00078a20 msgr2=0x7f8f0007aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f8f00078a20 0x7f8f0007aee0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8f040113e0 tx=0x7f8f0400b040 comp rx=0 tx=0).stop 2024-07-26T23:59:00.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 10 monclient: shutdown 2024-07-26T23:59:00.319 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8f141410b0 msgr2=0x7f8f1411ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 --2- 172.21.15.139:0/1871132579 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f8f141410b0 0x7f8f1411ead0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f8f0802b950 tx=0x7f8f080046c0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f13fff640 1 -- 172.21.15.139:0/1871132579 reap_dead start 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 shutdown_connections 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 >> 172.21.15.139:0/1871132579 conn(0x7f8f140b2f60 msgr2=0x7f8f14144220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 shutdown_connections 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 -- 172.21.15.139:0/1871132579 wait complete. 2024-07-26T23:59:00.320 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.301+0000 7f8f19fdf640 1 librados: shutdown 2024-07-26T23:59:00.329 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-07-26T23:59:00.431 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.416+0000 7fac92d18640 10 monclient: get_monmap_and_config 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.416+0000 7fac92d18640 10 monclient: build_initial_monmap 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.416+0000 7fac92d18640 10 monclient: monmap: 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:59:00.417633+0000 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:59:00.417633+0000 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:59:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:59:00.433 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 1 Processor -- start 2024-07-26T23:59:00.434 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 1 -- start start 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 10 monclient: init 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac8c1470c0 0x7fac8c140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 10 monclient(hunting): picked mon.noname-c con 0x7fac8c1470c0 addr v2:172.21.15.177:3300/0 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c140e90 0x7fac8c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.417+0000 7fac92d18640 10 monclient(hunting): picked mon.noname-b con 0x7fac8c140e90 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac8c0b4700 addr v1:172.21.15.139:6789/0 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac8c147ce0 con 0x7fac8c0b4700 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fac8c0acaf0 con 0x7fac8c1470c0 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fac8c1477c0 con 0x7fac8c140e90 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac92d18640 10 monclient(hunting): authenticate will time out at 2773.454045s 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac8bfff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c140e90 0x7fac8c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.418+0000 7fac90ab4640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac8c1470c0 0x7fac8c140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c140e90 0x7fac8c0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:33698/0 (socket says 172.21.15.139:33698) 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 1 -- 172.21.15.139:0/2483606686 learned_addr learned my addr 172.21.15.139:0/2483606686 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 10 monclient(hunting): get_auth_request con 0x7fac8c140e90 auth_method 0 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.419+0000 7fac8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.420+0000 7fac90ab4640 10 monclient(hunting): get_auth_request con 0x7fac8c1470c0 auth_method 0 2024-07-26T23:59:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.420+0000 7fac90ab4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.420+0000 7fac90ab4640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.420+0000 7fac90ab4640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 10 monclient(hunting): handle_auth_done global_id 16874 payload 274 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 1 -- 172.21.15.139:0/2483606686 >> v2:172.21.15.177:3300/0 conn(0x7fac8c1470c0 msgr2=0x7fac8c140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 1 --2- 172.21.15.139:0/2483606686 >> v2:172.21.15.177:3300/0 conn(0x7fac8c1470c0 0x7fac8c140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 1 -- 172.21.15.139:0/2483606686 >> v1:172.21.15.139:6789/0 conn(0x7fac8c0b4700 legacy=0x7fac8c1438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 1 monclient: found mon.noname-b 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:59:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.421+0000 7fac8bfff640 1 -- 172.21.15.139:0/2483606686 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8c0b4b90 con 0x7fac8c140e90 2024-07-26T23:59:00.439 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.423+0000 7fac8bfff640 1 --2- 172.21.15.139:0/2483606686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c140e90 0x7fac8c0b41c0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7fac78009870 tx=0x7fac7802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c91a31c88917d2a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 1 -- 172.21.15.139:0/2483606686 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac780317c0 con 0x7fac8c140e90 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: dump: 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.442 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: _finish_auth 0 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: _check_auth_tickets 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 1 -- 172.21.15.139:0/2483606686 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac78037070 con 0x7fac8c140e90 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac92d18640 5 monclient: authenticate success, global_id 16874 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac92d18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 1 -- 172.21.15.139:0/2483606686 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78031b80 con 0x7fac8c140e90 2024-07-26T23:59:00.443 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac92d18640 10 monclient: get_monmap_and_config success 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac92d18640 4 set_mon_vals no callback set 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.426+0000 7fac8b7fe640 10 monclient: dump: 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.444 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.445 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 10 monclient: shutdown 2024-07-26T23:59:00.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:00.447 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 -- 172.21.15.139:0/2483606686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c140e90 msgr2=0x7fac8c0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 --2- 172.21.15.139:0/2483606686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c140e90 0x7fac8c0b41c0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7fac78009870 tx=0x7fac7802b860 comp rx=0 tx=0).stop 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 -- 172.21.15.139:0/2483606686 shutdown_connections 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 --2- 172.21.15.139:0/2483606686 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c140e90 0x7fac8c0b41c0 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 --2- 172.21.15.139:0/2483606686 >> v2:172.21.15.177:3300/0 conn(0x7fac8c1470c0 0x7fac8c140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 -- 172.21.15.139:0/2483606686 >> 172.21.15.139:0/2483606686 conn(0x7fac8c0b62c0 msgr2=0x7fac8c0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 -- 172.21.15.139:0/2483606686 shutdown_connections 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.427+0000 7fac92d18640 1 -- 172.21.15.139:0/2483606686 wait complete. 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.428+0000 7fac92d18640 10 monclient: build_initial_monmap 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.428+0000 7fac92d18640 10 monclient: monmap: 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:59:00.448 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 librados: starting msgr at 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 librados: starting objecter 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 Processor -- start 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 -- start start 2024-07-26T23:59:00.449 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 librados: setting wanted keys 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 librados: calling monclient init 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 10 monclient: init 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c0b4700 0x7fac8c143620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 10 monclient(hunting): picked mon.noname-b con 0x7fac8c0b4700 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac8c140e90 0x7fac8c1cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.429+0000 7fac92d18640 10 monclient(hunting): picked mon.noname-a con 0x7fac8c140e90 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c1470c0 0x7fac8c1cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 10 monclient(hunting): picked mon.noname-c con 0x7fac8c1470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.450 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c074850 con 0x7fac8c140e90 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c0699f0 con 0x7fac8c0b4700 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fac8c141410 con 0x7fac8c1470c0 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac92d18640 10 monclient(hunting): authenticate will time out at 2773.466039s 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac912b5640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c1470c0 0x7fac8c1cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac90ab4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c0b4700 0x7fac8c143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac912b5640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c1470c0 0x7fac8c1cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:33712/0 (socket says 172.21.15.139:33712) 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac912b5640 1 -- 172.21.15.139:0/3440900625 learned_addr learned my addr 172.21.15.139:0/3440900625 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac912b5640 10 monclient(hunting): get_auth_request con 0x7fac8c1470c0 auth_method 0 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac912b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.451 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac912b5640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac912b5640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.430+0000 7fac90ab4640 10 monclient(hunting): get_auth_request con 0x7fac8c0b4700 auth_method 0 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac90ab4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac90ab4640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac90ab4640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac912b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac912b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac912b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac90ab4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac90ab4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:00.452 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.431+0000 7fac90ab4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:00.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 10 monclient(hunting): handle_auth_done global_id 20356 payload 824 2024-07-26T23:59:00.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 1 -- 172.21.15.139:0/3440900625 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c1470c0 msgr2=0x7fac8c1cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c1470c0 0x7fac8c1cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 1 -- 172.21.15.139:0/3440900625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac8c140e90 msgr2=0x7fac8c1cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac8c140e90 0x7fac8c1cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.453 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 10 monclient: _finish_hunting 0 2024-07-26T23:59:00.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 1 monclient: found mon.noname-b 2024-07-26T23:59:00.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:00.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:59:00.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.434+0000 7fac90ab4640 1 -- 172.21.15.139:0/3440900625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8c1d02d0 con 0x7fac8c0b4700 2024-07-26T23:59:00.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.437+0000 7fac90ab4640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c0b4700 0x7fac8c143620 secure :-1 s=READY pgs=4951 cs=0 l=1 rev1=1 crypto rx=0x7fac8000d520 tx=0x7fac8000d9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:00.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.437+0000 7fac897fa640 10 client.?.objecter ms_handle_connect 0x7fac8c0b4700 2024-07-26T23:59:00.454 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.438+0000 7fac897fa640 10 client.?.objecter resend_mon_ops 2024-07-26T23:59:00.460 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 1 -- 172.21.15.139:0/3440900625 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac800125f0 con 0x7fac8c0b4700 2024-07-26T23:59:00.460 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: dump: 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.461 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: _finish_auth 0 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: _check_auth_tickets 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 1 -- 172.21.15.139:0/3440900625 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac80019070 con 0x7fac8c0b4700 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 1 -- 172.21.15.139:0/3440900625 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac800129b0 con 0x7fac8c0b4700 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 5 monclient: authenticate success, global_id 20356 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 10 monclient: _renew_subs 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8c1cf6e0 con 0x7fac8c0b4700 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac897fa640 10 monclient: dump: 2024-07-26T23:59:00.462 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac8a7fc640 4 set_mon_vals no callback set 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 10 client.20356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 10 monclient: _renew_subs 2024-07-26T23:59:00.463 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:00.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac8c1d0820 con 0x7fac8c0b4700 2024-07-26T23:59:00.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.444+0000 7fac92d18640 1 librados: init done 2024-07-26T23:59:00.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.445+0000 7fac92d18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:00.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.445+0000 7fac92d18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:00.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.445+0000 7fac92d18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:00.464 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.445+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac8c0b41c0 con 0x7fac8c0b4700 2024-07-26T23:59:00.480 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.464+0000 7fac897fa640 1 -- 172.21.15.139:0/3440900625 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fac80012b50 con 0x7fac8c0b4700 2024-07-26T23:59:00.480 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.465+0000 7fac897fa640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fac700789a0 0x7fac7007ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:00.480 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.465+0000 7fac8bfff640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fac700789a0 0x7fac7007ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:00.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.465+0000 7fac8bfff640 10 monclient: get_auth_request con 0x7fac700789a0 auth_method 0 2024-07-26T23:59:00.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.466+0000 7fac897fa640 10 client.20356.objecter ms_handle_connect 0x7fac700789a0 2024-07-26T23:59:00.481 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.466+0000 7fac8bfff640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fac700789a0 0x7fac7007ae60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fac78036000 tx=0x7fac78030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:00.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.468+0000 7fac897fa640 1 -- 172.21.15.139:0/3440900625 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 501..970) ==== 6365+0+0 (secure 0 0 0) 0x7fac80012e40 con 0x7fac8c0b4700 2024-07-26T23:59:00.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.468+0000 7fac897fa640 10 client.20356.objecter ms_dispatch 0x7fac8c000b90 osd_map(970..970 src has 501..970) 2024-07-26T23:59:00.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.468+0000 7fac897fa640 3 client.20356.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:59:00.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.468+0000 7fac897fa640 3 client.20356.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:59:00.484 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.468+0000 7fac897fa640 20 client.20356.objecter dump_active .. 0 homeless 2024-07-26T23:59:00.525 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:08.182 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.534 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:08.192 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.561 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:08.219 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.603 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:08.261 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.606 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:08.264 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.629 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:08.286 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.705+0000 7fac897fa640 1 -- 172.21.15.139:0/3440900625 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fac800627e0 con 0x7fac8c0b4700 2024-07-26T23:59:00.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.705+0000 7fac897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:00.720 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.705+0000 7fac897fa640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:59:00.788 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:08.437 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.789 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:08.438 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.869 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:08.518 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:00.928 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.913+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fac8c1d0c40 con 0x7fac700789a0 2024-07-26T23:59:00.929 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.913+0000 7fac897fa640 1 -- 172.21.15.139:0/3440900625 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fac8c1d0c40 con 0x7fac700789a0 2024-07-26T23:59:00.929 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 6 to deep-scrub 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 10 librados: watch_flush enter 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 10 librados: watch_flush exit 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 20 client.20356.objecter shutdown clearing up homeless session... 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 10 client.20356.objecter successfully canceled tick 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fac700789a0 msgr2=0x7fac7007ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fac700789a0 0x7fac7007ae60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fac78036000 tx=0x7fac78030000 comp rx=0 tx=0).stop 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 10 monclient: shutdown 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:00.932 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c0b4700 msgr2=0x7fac8c143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c0b4700 0x7fac8c143620 secure :-1 s=READY pgs=4951 cs=0 l=1 rev1=1 crypto rx=0x7fac8000d520 tx=0x7fac8000d9e0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 shutdown_connections 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fac700789a0 0x7fac7007ae60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fac8c1470c0 0x7fac8c1cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac8c140e90 0x7fac8c1cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 --2- 172.21.15.139:0/3440900625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c0b4700 0x7fac8c143620 unknown :-1 s=CLOSED pgs=4951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 >> 172.21.15.139:0/3440900625 conn(0x7fac8c0b62c0 msgr2=0x7fac8c0b73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 shutdown_connections 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 -- 172.21.15.139:0/3440900625 wait complete. 2024-07-26T23:59:00.933 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:00.916+0000 7fac92d18640 1 librados: shutdown 2024-07-26T23:59:00.947 INFO:tasks.ceph:Scrubbing osd.7 2024-07-26T23:59:00.947 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-07-26T23:59:01.061 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.046+0000 7fd2e011c640 10 monclient: get_monmap_and_config 2024-07-26T23:59:01.061 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.046+0000 7fd2e011c640 10 monclient: build_initial_monmap 2024-07-26T23:59:01.061 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.046+0000 7fd2e011c640 10 monclient: monmap: 2024-07-26T23:59:01.061 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:59:01.047220+0000 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:59:01.047220+0000 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:59:01.062 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:59:01.063 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.046+0000 7fd2e011c640 1 Processor -- start 2024-07-26T23:59:01.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 1 -- start start 2024-07-26T23:59:01.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient: init 2024-07-26T23:59:01.064 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd2d806e2b0 0x7fd2d8140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d806e2b0 addr v2:172.21.15.177:3300/0 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fd2d80749f0 0x7fd2d80b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d80749f0 addr v2:172.21.15.139:3300/0 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2d81411e0 addr v1:172.21.15.177:6789/0 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d806d830 con 0x7fd2d81411e0 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d806e690 con 0x7fd2d80749f0 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d806f370 con 0x7fd2d806e2b0 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.047+0000 7fd2e011c640 10 monclient(hunting): authenticate will time out at 2774.083656s 2024-07-26T23:59:01.065 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.049+0000 7fd2dd6b7640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fd2d80749f0 0x7fd2d80b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.050+0000 7fd2dd6b7640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fd2d80749f0 0x7fd2d80b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:51538/0 (socket says 172.21.15.139:51538) 2024-07-26T23:59:01.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.050+0000 7fd2dd6b7640 1 -- 172.21.15.139:0/328595651 learned_addr learned my addr 172.21.15.139:0/328595651 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:01.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.050+0000 7fd2dd6b7640 10 monclient(hunting): get_auth_request con 0x7fd2d80749f0 auth_method 0 2024-07-26T23:59:01.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.050+0000 7fd2dd6b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.050+0000 7fd2dd6b7640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.066 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.050+0000 7fd2dd6b7640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.067 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.052+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:01.067 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.052+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:01.067 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.052+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:01.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_done global_id 15609 payload 274 2024-07-26T23:59:01.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 1 -- 172.21.15.139:0/328595651 >> v2:172.21.15.177:3300/0 conn(0x7fd2d806e2b0 msgr2=0x7fd2d8140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:01.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 1 --2- 172.21.15.139:0/328595651 >> v2:172.21.15.177:3300/0 conn(0x7fd2d806e2b0 0x7fd2d8140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 1 -- 172.21.15.139:0/328595651 >> v1:172.21.15.177:6789/0 conn(0x7fd2d81411e0 legacy=0x7fd2d80b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:01.070 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 10 monclient: _finish_hunting 0 2024-07-26T23:59:01.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 1 monclient: found mon.noname-a 2024-07-26T23:59:01.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:01.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:59:01.071 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.055+0000 7fd2dd6b7640 1 -- 172.21.15.139:0/328595651 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d806e840 con 0x7fd2d80749f0 2024-07-26T23:59:01.075 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.060+0000 7fd2dd6b7640 1 --2- 172.21.15.139:0/328595651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d80749f0 0x7fd2d80b0b70 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80b8740 tx=0x7fd2cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a2717d690a5f80 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.086 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 1 -- 172.21.15.139:0/328595651 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc035020 con 0x7fd2d80749f0 2024-07-26T23:59:01.086 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.086 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: dump: 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: _finish_auth 0 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: _check_auth_tickets 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 1 -- 172.21.15.139:0/328595651 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2cc031870 con 0x7fd2d80749f0 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2e011c640 5 monclient: authenticate success, global_id 15609 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2e011c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2e011c640 10 monclient: get_monmap_and_config success 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2e011c640 4 set_mon_vals no callback set 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 1 -- 172.21.15.139:0/328595651 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc031c40 con 0x7fd2d80749f0 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:59:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.071+0000 7fd2dceb6640 10 monclient: dump: 2024-07-26T23:59:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.090 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.090 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.090 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.090 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.090 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 10 monclient: shutdown 2024-07-26T23:59:01.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:01.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 -- 172.21.15.139:0/328595651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d80749f0 msgr2=0x7fd2d80b0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 --2- 172.21.15.139:0/328595651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d80749f0 0x7fd2d80b0b70 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80b8740 tx=0x7fd2cc02b910 comp rx=0 tx=0).stop 2024-07-26T23:59:01.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 -- 172.21.15.139:0/328595651 shutdown_connections 2024-07-26T23:59:01.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 --2- 172.21.15.139:0/328595651 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d80749f0 0x7fd2d80b0b70 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.093 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 --2- 172.21.15.139:0/328595651 >> v2:172.21.15.177:3300/0 conn(0x7fd2d806e2b0 0x7fd2d8140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 -- 172.21.15.139:0/328595651 >> 172.21.15.139:0/328595651 conn(0x7fd2d80b7ac0 msgr2=0x7fd2d80b7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 -- 172.21.15.139:0/328595651 shutdown_connections 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.072+0000 7fd2e011c640 1 -- 172.21.15.139:0/328595651 wait complete. 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 10 monclient: build_initial_monmap 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 10 monclient: monmap: 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:01.094 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 1 librados: starting msgr at 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 1 librados: starting objecter 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 1 Processor -- start 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 1 -- start start 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 1 librados: setting wanted keys 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 1 librados: calling monclient init 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.073+0000 7fd2e011c640 10 monclient: init 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient: _add_conns ranks=[0,2,1] 2024-07-26T23:59:01.095 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d806e2b0 0x7fd2d813d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d806e2b0 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd2d80749f0 0x7fd2d8135a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d80749f0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2d81411e0 0x7fd2d8135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d81411e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d814e3b0 con 0x7fd2d806e2b0 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d806f520 con 0x7fd2d81411e0 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d814e530 con 0x7fd2d80749f0 2024-07-26T23:59:01.096 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:01.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.074+0000 7fd2e011c640 10 monclient(hunting): authenticate will time out at 2774.110450s 2024-07-26T23:59:01.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.077+0000 7fd2ddeb8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d806e2b0 0x7fd2d813d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.077+0000 7fd2dd6b7640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd2d80749f0 0x7fd2d8135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.080+0000 7fd2ddeb8640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d806e2b0 0x7fd2d813d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:51548/0 (socket says 172.21.15.139:51548) 2024-07-26T23:59:01.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.080+0000 7fd2ddeb8640 1 -- 172.21.15.139:0/4177486616 learned_addr learned my addr 172.21.15.139:0/4177486616 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:01.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.080+0000 7fd2ddeb8640 10 monclient(hunting): get_auth_request con 0x7fd2d806e2b0 auth_method 0 2024-07-26T23:59:01.097 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.080+0000 7fd2ddeb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.080+0000 7fd2ddeb8640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.098 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.080+0000 7fd2ddeb8640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.084+0000 7fd2dd6b7640 10 monclient(hunting): get_auth_request con 0x7fd2d80749f0 auth_method 0 2024-07-26T23:59:01.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.084+0000 7fd2dd6b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.084+0000 7fd2dd6b7640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.099 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.084+0000 7fd2dd6b7640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.085+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:01.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.085+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:01.100 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.085+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:01.104 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 10 monclient(hunting): handle_auth_done global_id 16880 payload 824 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 1 -- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2d81411e0 msgr2=0x7fd2d8135fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2d81411e0 0x7fd2d8135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 1 -- 172.21.15.139:0/4177486616 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d806e2b0 msgr2=0x7fd2d813d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2d806e2b0 0x7fd2d813d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 10 monclient: _finish_hunting 0 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 1 monclient: found mon.noname-c 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.139:3301/0 2024-07-26T23:59:01.105 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.089+0000 7fd2dd6b7640 1 -- 172.21.15.139:0/4177486616 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d80717b0 con 0x7fd2d80749f0 2024-07-26T23:59:01.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.093+0000 7fd2dd6b7640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd2d80749f0 0x7fd2d8135a70 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc033000 tx=0x7fd2cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.093+0000 7fd2d67fc640 10 client.?.objecter ms_handle_connect 0x7fd2d80749f0 2024-07-26T23:59:01.108 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.093+0000 7fd2d67fc640 10 client.?.objecter resend_mon_ops 2024-07-26T23:59:01.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0443f0 con 0x7fd2d80749f0 2024-07-26T23:59:01.114 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: dump: 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.115 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: _finish_auth 0 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: _check_auth_tickets 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2cc044590 con 0x7fd2d80749f0 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 5 monclient: authenticate success, global_id 16880 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 10 monclient: _renew_subs 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d813e1a0 con 0x7fd2d80749f0 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc044980 con 0x7fd2d80749f0 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.116 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d67fc640 10 monclient: dump: 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 10 client.16880.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 10 monclient: _renew_subs 2024-07-26T23:59:01.117 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:59:01.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d8071d00 con 0x7fd2d80749f0 2024-07-26T23:59:01.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2e011c640 1 librados: init done 2024-07-26T23:59:01.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.099+0000 7fd2d7fff640 4 set_mon_vals no callback set 2024-07-26T23:59:01.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.100+0000 7fd2e011c640 10 client.16880.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:01.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.100+0000 7fd2e011c640 15 client.16880.objecter _session_command_op_assign -1 1 2024-07-26T23:59:01.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.100+0000 7fd2e011c640 20 client.16880.objecter _get_session osd=-1 returning homeless 2024-07-26T23:59:01.118 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.100+0000 7fd2e011c640 10 client.16880.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:01.119 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.100+0000 7fd2e011c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.139:3301/0 2024-07-26T23:59:01.119 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.100+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2d8140ca0 con 0x7fd2d80749f0 2024-07-26T23:59:01.138 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.123+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 <== mon.2 v2:172.21.15.139:3301/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fd2cc04b060 con 0x7fd2d80749f0 2024-07-26T23:59:01.139 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.123+0000 7fd2d67fc640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd2c40789f0 0x7fd2c407aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.139 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.123+0000 7fd2ddeb8640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd2c40789f0 0x7fd2c407aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.139 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.123+0000 7fd2ddeb8640 10 monclient: get_auth_request con 0x7fd2c40789f0 auth_method 0 2024-07-26T23:59:01.139 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.124+0000 7fd2ddeb8640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd2c40789f0 0x7fd2c407aeb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009f50 tx=0x7fd2c80095b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.124+0000 7fd2d67fc640 10 client.16880.objecter ms_handle_connect 0x7fd2c40789f0 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 <== mon.2 v2:172.21.15.139:3301/0 5 ==== osd_map(970..970 src has 735..970) ==== 6365+0+0 (secure 0 0 0) 0x7fd2cc031750 con 0x7fd2d80749f0 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 10 client.16880.objecter ms_dispatch 0x7fd2d8000b90 osd_map(970..970 src has 735..970) 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 3 client.16880.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 3 client.16880.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 10 client.16880.objecter checking command 1 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7fd2c4080770 0x7fd2c4082c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 20 client.16880.objecter _get_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.143 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 20 client.16880.objecter put_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 15 client.16880.objecter _session_command_op_remove -1 1 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 20 client.16880.objecter _get_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 20 client.16880.objecter get_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 15 client.16880.objecter _session_command_op_assign 7 1 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 20 client.16880.objecter put_session s=0x7fd2c40800a0 osd=7 4 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 10 client.16880.objecter _send_command 1 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 --> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2cc045e50 con 0x7fd2c4080770 2024-07-26T23:59:01.144 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.127+0000 7fd2d67fc640 20 client.16880.objecter dump_active .. 0 homeless 2024-07-26T23:59:01.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.128+0000 7fd2de6b9640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7fd2c4080770 0x7fd2c4082c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.145 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.128+0000 7fd2de6b9640 10 monclient: get_auth_request con 0x7fd2c4080770 auth_method 0 2024-07-26T23:59:01.146 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.131+0000 7fd2de6b9640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7fd2c4080770 0x7fd2c4082c10 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.146 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.131+0000 7fd2d67fc640 10 client.16880.objecter ms_handle_connect 0x7fd2c4080770 2024-07-26T23:59:01.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.158+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 <== osd.7 v2:172.21.15.177:6818/3299104561 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2cc045e50 con 0x7fd2c4080770 2024-07-26T23:59:01.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.158+0000 7fd2d67fc640 10 client.16880.objecter ms_dispatch 0x7fd2d8000b90 command_reply(tid 1: 0 ) 2024-07-26T23:59:01.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.158+0000 7fd2d67fc640 10 client.16880.objecter _finish_command 1 = system:0 2024-07-26T23:59:01.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.158+0000 7fd2d67fc640 20 client.16880.objecter put_session s=0x7fd2c40800a0 osd=7 4 2024-07-26T23:59:01.174 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.158+0000 7fd2d67fc640 15 client.16880.objecter _session_command_op_remove 7 1 2024-07-26T23:59:01.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 10 client.16880.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-07-26T23:59:01.187 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 15 client.16880.objecter _session_command_op_assign -1 2 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 20 client.16880.objecter _get_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 20 client.16880.objecter put_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 20 client.16880.objecter _get_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 15 client.16880.objecter _session_command_op_remove -1 2 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 20 client.16880.objecter get_session s=0x7fd2c40800a0 osd=7 3 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 15 client.16880.objecter _session_command_op_assign 7 2 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 20 client.16880.objecter put_session s=0x7fd2c40800a0 osd=7 4 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 10 client.16880.objecter _send_command 2 2024-07-26T23:59:01.188 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.172+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 --> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd2d8134460 con 0x7fd2c4080770 2024-07-26T23:59:01.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2d67fc640 1 -- 172.21.15.139:0/4177486616 <== osd.7 v2:172.21.15.177:6818/3299104561 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd2d8134460 con 0x7fd2c4080770 2024-07-26T23:59:01.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2d67fc640 10 client.16880.objecter ms_dispatch 0x7fd2d8000b90 command_reply(tid 2: 0 ) 2024-07-26T23:59:01.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2d67fc640 10 client.16880.objecter _finish_command 2 = system:0 2024-07-26T23:59:01.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2d67fc640 20 client.16880.objecter put_session s=0x7fd2c40800a0 osd=7 4 2024-07-26T23:59:01.203 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2d67fc640 15 client.16880.objecter _session_command_op_remove 7 2 2024-07-26T23:59:01.204 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-07-26T23:59:01.204 INFO:teuthology.orchestra.run.smithi139.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T23:59:01.204 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-07-26T23:59:01.205 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 10 librados: watch_flush enter 2024-07-26T23:59:01.205 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 10 librados: watch_flush exit 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 10 client.16880.objecter close_session for osd.7 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7fd2c4080770 msgr2=0x7fd2c4082c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6818/3299104561,v1:172.21.15.177:6819/3299104561] conn(0x7fd2c4080770 0x7fd2c4082c10 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 20 client.16880.objecter put_session s=0x7fd2c40800a0 osd=7 1 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 20 client.16880.objecter shutdown clearing up homeless session... 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 10 client.16880.objecter successfully canceled tick 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd2c40789f0 msgr2=0x7fd2c407aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fd2c40789f0 0x7fd2c407aeb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009f50 tx=0x7fd2c80095b0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 10 monclient: shutdown 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 20 monclient: shutdown canceling and discarding version request 1 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd2d80749f0 msgr2=0x7fd2d8135a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.206 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 --2- 172.21.15.139:0/4177486616 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fd2d80749f0 0x7fd2d8135a70 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc033000 tx=0x7fd2cc0046c0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2de6b9640 1 -- 172.21.15.139:0/4177486616 reap_dead start 2024-07-26T23:59:01.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 shutdown_connections 2024-07-26T23:59:01.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.188+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 >> 172.21.15.139:0/4177486616 conn(0x7fd2d80b7ac0 msgr2=0x7fd2d80b4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:01.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.189+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 shutdown_connections 2024-07-26T23:59:01.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.189+0000 7fd2e011c640 1 -- 172.21.15.139:0/4177486616 wait complete. 2024-07-26T23:59:01.207 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.189+0000 7fd2e011c640 1 librados: shutdown 2024-07-26T23:59:01.218 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-07-26T23:59:01.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.308+0000 7fbcc1a5e640 10 monclient: get_monmap_and_config 2024-07-26T23:59:01.323 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.308+0000 7fbcc1a5e640 10 monclient: build_initial_monmap 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.308+0000 7fbcc1a5e640 10 monclient: monmap: 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:59:01.309523+0000 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:59:01.309523+0000 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:59:01.324 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:59:01.325 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.309+0000 7fbcc1a5e640 1 Processor -- start 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.309+0000 7fbcc1a5e640 1 -- start start 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.309+0000 7fbcc1a5e640 10 monclient: init 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.309+0000 7fbcc1a5e640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.309+0000 7fbcc1a5e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fbcbc1470c0 0x7fbcbc140950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 10 monclient(hunting): picked mon.noname-a con 0x7fbcbc1470c0 addr v2:172.21.15.139:3300/0 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc140e90 0x7fbcbc0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 10 monclient(hunting): picked mon.noname-b con 0x7fbcbc140e90 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:01.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbcbc0b4700 addr v1:172.21.15.139:6789/0 2024-07-26T23:59:01.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcbc147ce0 con 0x7fbcbc0b4700 2024-07-26T23:59:01.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fbcbc0acaf0 con 0x7fbcbc1470c0 2024-07-26T23:59:01.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fbcbc1477c0 con 0x7fbcbc140e90 2024-07-26T23:59:01.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:01.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.310+0000 7fbcc1a5e640 10 monclient(hunting): authenticate will time out at 2774.346110s 2024-07-26T23:59:01.331 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.316+0000 7fbcba7fc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc140e90 0x7fbcbc0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.317+0000 7fbcba7fc640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc140e90 0x7fbcbc0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3301/0 says I am v2:172.21.15.139:33730/0 (socket says 172.21.15.139:33730) 2024-07-26T23:59:01.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.317+0000 7fbcba7fc640 1 -- 172.21.15.139:0/3677197930 learned_addr learned my addr 172.21.15.139:0/3677197930 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:01.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.317+0000 7fbcba7fc640 10 monclient(hunting): get_auth_request con 0x7fbcbc140e90 auth_method 0 2024-07-26T23:59:01.332 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.317+0000 7fbcba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.317+0000 7fbcba7fc640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.333 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.317+0000 7fbcba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.319+0000 7fbcba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:01.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.319+0000 7fbcba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:01.334 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.319+0000 7fbcba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:01.335 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.320+0000 7fbcbaffd640 1 --2- 172.21.15.139:0/3677197930 >> v2:172.21.15.139:3300/0 conn(0x7fbcbc1470c0 0x7fbcbc140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.321+0000 7fbcba7fc640 10 monclient(hunting): handle_auth_done global_id 16883 payload 274 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 1 -- 172.21.15.139:0/3677197930 >> v2:172.21.15.139:3300/0 conn(0x7fbcbc1470c0 msgr2=0x7fbcbc140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 1 --2- 172.21.15.139:0/3677197930 >> v2:172.21.15.139:3300/0 conn(0x7fbcbc1470c0 0x7fbcbc140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 1 -- 172.21.15.139:0/3677197930 >> v1:172.21.15.139:6789/0 conn(0x7fbcbc0b4700 legacy=0x7fbcbc1438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 10 monclient: _finish_hunting 0 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 1 monclient: found mon.noname-b 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:59:01.337 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.322+0000 7fbcba7fc640 1 -- 172.21.15.139:0/3677197930 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcbc0b4b90 con 0x7fbcbc140e90 2024-07-26T23:59:01.341 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.326+0000 7fbcba7fc640 1 --2- 172.21.15.139:0/3677197930 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc140e90 0x7fbcbc0b41c0 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009870 tx=0x7fbcb002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5de561258532097 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 1 -- 172.21.15.139:0/3677197930 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00317c0 con 0x7fbcbc140e90 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: dump: 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.349 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: _finish_auth 0 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: _check_auth_tickets 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 1 -- 172.21.15.139:0/3677197930 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb0037070 con 0x7fbcbc140e90 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 1 -- 172.21.15.139:0/3677197930 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0031b80 con 0x7fbcbc140e90 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 5 monclient: authenticate success, global_id 16883 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 10 monclient: get_monmap_and_config success 2024-07-26T23:59:01.350 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 4 set_mon_vals no callback set 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcb9ffb640 10 monclient: dump: 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.351 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.352 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.352 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.352 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 10 monclient: shutdown 2024-07-26T23:59:01.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:01.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3677197930 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc140e90 msgr2=0x7fbcbc0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3677197930 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc140e90 0x7fbcbc0b41c0 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009870 tx=0x7fbcb002b860 comp rx=0 tx=0).stop 2024-07-26T23:59:01.354 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3677197930 shutdown_connections 2024-07-26T23:59:01.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3677197930 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc140e90 0x7fbcbc0b41c0 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3677197930 >> v2:172.21.15.139:3300/0 conn(0x7fbcbc1470c0 0x7fbcbc140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.334+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3677197930 >> 172.21.15.139:0/3677197930 conn(0x7fbcbc0b62c0 msgr2=0x7fbcbc0b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:01.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.335+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3677197930 shutdown_connections 2024-07-26T23:59:01.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.335+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3677197930 wait complete. 2024-07-26T23:59:01.355 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.335+0000 7fbcc1a5e640 10 monclient: build_initial_monmap 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.335+0000 7fbcc1a5e640 10 monclient: monmap: 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:59:01.356 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.335+0000 7fbcc1a5e640 1 librados: starting msgr at 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.335+0000 7fbcc1a5e640 1 librados: starting objecter 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 Processor -- start 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 -- start start 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 librados: setting wanted keys 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 librados: calling monclient init 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient: init 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcbc0b4700 0x7fbcbc143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): picked mon.noname-b con 0x7fbcbc0b4700 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcbc140e90 0x7fbcbc0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.357 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): picked mon.noname-a con 0x7fbcbc140e90 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc1470c0 0x7fbcbc1d8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): picked mon.noname-c con 0x7fbcbc1470c0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fbcbc074850 con 0x7fbcbc140e90 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbcbc0699f0 con 0x7fbcbc0b4700 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fbcbc141410 con 0x7fbcbc1470c0 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.336+0000 7fbcc1a5e640 10 monclient(hunting): authenticate will time out at 2774.372881s 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.338+0000 7fbcbaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcbc0b4700 0x7fbcbc143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.338+0000 7fbcbaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcbc0b4700 0x7fbcbc143aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:47632/0 (socket says 172.21.15.139:47632) 2024-07-26T23:59:01.358 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.338+0000 7fbcbaffd640 1 -- 172.21.15.139:0/3260551784 learned_addr learned my addr 172.21.15.139:0/3260551784 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.338+0000 7fbcbaffd640 10 monclient(hunting): get_auth_request con 0x7fbcbc0b4700 auth_method 0 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.338+0000 7fbcbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.338+0000 7fbcbaffd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.338+0000 7fbcbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.339+0000 7fbcbb7fe640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc1470c0 0x7fbcbc1d8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.339+0000 7fbcbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.339+0000 7fbcbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:01.359 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.339+0000 7fbcbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:01.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.340+0000 7fbcbb7fe640 10 monclient(hunting): get_auth_request con 0x7fbcbc1470c0 auth_method 0 2024-07-26T23:59:01.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.340+0000 7fbcbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.340+0000 7fbcbb7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.340+0000 7fbcbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.341+0000 7fbcba7fc640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcbc140e90 0x7fbcbc0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 10 monclient(hunting): handle_auth_done global_id 20359 payload 824 2024-07-26T23:59:01.360 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 1 -- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc1470c0 msgr2=0x7fbcbc1d8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc1470c0 0x7fbcbc1d8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 1 -- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcbc140e90 msgr2=0x7fbcbc0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcbc140e90 0x7fbcbc0aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 10 monclient: _finish_hunting 0 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 1 monclient: found mon.noname-b 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:59:01.361 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.342+0000 7fbcbaffd640 1 -- 172.21.15.139:0/3260551784 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcbc0b0de0 con 0x7fbcbc0b4700 2024-07-26T23:59:01.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.346+0000 7fbcbaffd640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcbc0b4700 0x7fbcbc143aa0 secure :-1 s=READY pgs=4952 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00ccd0 tx=0x7fbcac00a5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.346+0000 7fbca3fff640 10 client.?.objecter ms_handle_connect 0x7fbcbc0b4700 2024-07-26T23:59:01.362 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.346+0000 7fbca3fff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:59:01.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbca3fff640 1 -- 172.21.15.139:0/3260551784 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac003990 con 0x7fbcbc0b4700 2024-07-26T23:59:01.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbca3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:59:01.364 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbca3fff640 10 monclient: dump: 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbca3fff640 10 monclient: _finish_auth 0 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbca3fff640 10 monclient: _check_auth_tickets 2024-07-26T23:59:01.365 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 5 monclient: authenticate success, global_id 20359 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 10 monclient: _renew_subs 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcbc1d6920 con 0x7fbcbc0b4700 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 10 client.20359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 10 monclient: _renew_subs 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcbc1d9500 con 0x7fbcbc0b4700 2024-07-26T23:59:01.366 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.349+0000 7fbcc1a5e640 1 librados: init done 2024-07-26T23:59:01.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbcc1a5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:01.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbcc1a5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:01.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbcc1a5e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:01.367 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcbc0af5b0 con 0x7fbcbc0b4700 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbca3fff640 1 -- 172.21.15.139:0/3260551784 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcac022480 con 0x7fbcbc0b4700 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbca3fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbca3fff640 1 -- 172.21.15.139:0/3260551784 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac01d060 con 0x7fbcbc0b4700 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbca3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbca3fff640 10 monclient: dump: 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.368 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.369 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.369 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.369 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.350+0000 7fbcb97fa640 4 set_mon_vals no callback set 2024-07-26T23:59:01.494 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:09.151 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.495 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:09.152 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.507 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:09.164 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.567 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:09.225 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.569 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:09.226 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.582 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:09.240 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.600+0000 7fbca3fff640 1 -- 172.21.15.139:0/3260551784 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fbcac022830 con 0x7fbcbc0b4700 2024-07-26T23:59:01.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.600+0000 7fbca3fff640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fbca80789d0 0x7fbca807ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.601+0000 7fbcba7fc640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fbca80789d0 0x7fbca807ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.601+0000 7fbcba7fc640 10 monclient: get_auth_request con 0x7fbca80789d0 auth_method 0 2024-07-26T23:59:01.616 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.601+0000 7fbcba7fc640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fbca80789d0 0x7fbca807ae90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0036000 tx=0x7fbcb0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.617 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.601+0000 7fbca3fff640 10 client.20359.objecter ms_handle_connect 0x7fbca80789d0 2024-07-26T23:59:01.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.604+0000 7fbca3fff640 1 -- 172.21.15.139:0/3260551784 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 735..970) ==== 6365+0+0 (secure 0 0 0) 0x7fbcac0975b0 con 0x7fbcbc0b4700 2024-07-26T23:59:01.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.604+0000 7fbca3fff640 10 client.20359.objecter ms_dispatch 0x7fbcbc000b90 osd_map(970..970 src has 735..970) 2024-07-26T23:59:01.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.604+0000 7fbca3fff640 3 client.20359.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:59:01.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.604+0000 7fbca3fff640 3 client.20359.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:59:01.619 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.604+0000 7fbca3fff640 20 client.20359.objecter dump_active .. 0 homeless 2024-07-26T23:59:01.630 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.615+0000 7fbca3fff640 1 -- 172.21.15.139:0/3260551784 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fbcac014030 con 0x7fbcbc0b4700 2024-07-26T23:59:01.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.615+0000 7fbca3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:01.631 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.615+0000 7fbca3fff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:59:01.832 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.817+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fbcbc0b8bf0 con 0x7fbca80789d0 2024-07-26T23:59:01.833 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.818+0000 7fbca3fff640 1 -- 172.21.15.139:0/3260551784 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbcbc0b8bf0 con 0x7fbca80789d0 2024-07-26T23:59:01.833 INFO:teuthology.orchestra.run.smithi139.stderr:instructed osd(s) 7 to deep-scrub 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 10 librados: watch_flush enter 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 10 librados: watch_flush exit 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 20 client.20359.objecter shutdown clearing up homeless session... 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 10 client.20359.objecter successfully canceled tick 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fbca80789d0 msgr2=0x7fbca807ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fbca80789d0 0x7fbca807ae90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0036000 tx=0x7fbcb0008030 comp rx=0 tx=0).stop 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 10 monclient: shutdown 2024-07-26T23:59:01.836 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcbc0b4700 msgr2=0x7fbcbc143aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcbc0b4700 0x7fbcbc143aa0 secure :-1 s=READY pgs=4952 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00ccd0 tx=0x7fbcac00a5c0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 shutdown_connections 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fbca80789d0 0x7fbca807ae90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fbcbc1470c0 0x7fbcbc1d8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcbc140e90 0x7fbcbc0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 --2- 172.21.15.139:0/3260551784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcbc0b4700 0x7fbcbc143aa0 unknown :-1 s=CLOSED pgs=4952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 >> 172.21.15.139:0/3260551784 conn(0x7fbcbc0b62c0 msgr2=0x7fbcbc0b73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 shutdown_connections 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 -- 172.21.15.139:0/3260551784 wait complete. 2024-07-26T23:59:01.837 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.820+0000 7fbcc1a5e640 1 librados: shutdown 2024-07-26T23:59:01.847 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T23:59:01.902 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:09.551 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.903 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:09.552 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.912 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:09.561 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.932+0000 7f7031ddf640 10 monclient: get_monmap_and_config 2024-07-26T23:59:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.932+0000 7f7031ddf640 10 monclient: build_initial_monmap 2024-07-26T23:59:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.932+0000 7f7031ddf640 10 monclient: monmap: 2024-07-26T23:59:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:59:01.933288+0000 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:59:01.933288+0000 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:59:01.948 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.932+0000 7f7031ddf640 1 Processor -- start 2024-07-26T23:59:01.949 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 1 -- start start 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient: init 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f702c0b6560 0x7f702c0b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient(hunting): picked mon.noname-c con 0x7f702c0b6560 addr v2:172.21.15.177:3300/0 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f702c0b6940 addr v1:172.21.15.139:6789/0 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c13da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient(hunting): picked mon.noname-b con 0x7f702c13d600 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702c143e70 con 0x7f702c0b6940 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f702c0acc10 con 0x7f702c0b6560 2024-07-26T23:59:01.950 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f702c0ab6f0 con 0x7f702c13d600 2024-07-26T23:59:01.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:01.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.933+0000 7f7031ddf640 10 monclient(hunting): authenticate will time out at 2774.969784s 2024-07-26T23:59:01.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.935+0000 7f702bfff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c13da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.936+0000 7f702affd640 1 --1- >> v1:172.21.15.139:6789/0 conn(0x7f702c0b6940 0x7f702c144350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.139:6789/0 says I am v1:172.21.15.139:45552/0 (socket says 172.21.15.139:45552) 2024-07-26T23:59:01.951 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.936+0000 7f702affd640 1 -- 172.21.15.139:0/871416231 learned_addr learned my addr 172.21.15.139:0/871416231 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:59:01.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.937+0000 7f702bfff640 10 monclient(hunting): get_auth_request con 0x7f702c13d600 auth_method 0 2024-07-26T23:59:01.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.937+0000 7f702bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.937+0000 7f702bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.937+0000 7f702bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.953 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.938+0000 7f702b7fe640 1 --2- 172.21.15.139:0/871416231 >> v2:172.21.15.177:3300/0 conn(0x7f702c0b6560 0x7f702c0b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.939+0000 7f702bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:01.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.939+0000 7f702bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:01.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.939+0000 7f702bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:01.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.939+0000 7f702b7fe640 10 monclient(hunting): get_auth_request con 0x7f702c0b6560 auth_method 0 2024-07-26T23:59:01.955 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.939+0000 7f702b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.955 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.939+0000 7f702b7fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.955 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.939+0000 7f702b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 10 monclient(hunting): handle_auth_done global_id 16889 payload 274 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 1 -- 172.21.15.139:0/871416231 >> v2:172.21.15.177:3300/0 conn(0x7f702c0b6560 msgr2=0x7f702c0b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 1 --2- 172.21.15.139:0/871416231 >> v2:172.21.15.177:3300/0 conn(0x7f702c0b6560 0x7f702c0b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 1 -- 172.21.15.139:0/871416231 >> v1:172.21.15.139:6789/0 conn(0x7f702c0b6940 legacy=0x7f702c144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 10 monclient: _finish_hunting 0 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 1 monclient: found mon.noname-b 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:01.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.139:3301/0 2024-07-26T23:59:01.960 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.943+0000 7f702bfff640 1 -- 172.21.15.139:0/871416231 --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702c0b47a0 con 0x7f702c13d600 2024-07-26T23:59:01.963 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.948+0000 7f702bfff640 1 --2- 172.21.15.139:0/871416231 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c13da30 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f7024009d90 tx=0x7f702402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79aca0c2822bc4c6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 1 -- 172.21.15.139:0/871416231 <== mon.2 v2:172.21.15.139:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024039070 con 0x7f702c13d600 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: dump: 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.975 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: _finish_auth 0 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: _check_auth_tickets 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 1 -- 172.21.15.139:0/871416231 <== mon.2 v2:172.21.15.139:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f702402cbc0 con 0x7f702c13d600 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 1 -- 172.21.15.139:0/871416231 <== mon.2 v2:172.21.15.139:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024034440 con 0x7f702c13d600 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:01.976 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f702a7fc640 10 monclient: dump: 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:01.977 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:01.978 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f7031ddf640 5 monclient: authenticate success, global_id 16889 2024-07-26T23:59:01.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f7031ddf640 10 monclient: get_monmap_and_config success 2024-07-26T23:59:01.978 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.959+0000 7f7031ddf640 4 set_mon_vals no callback set 2024-07-26T23:59:01.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 10 monclient: shutdown 2024-07-26T23:59:01.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:01.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 -- 172.21.15.139:0/871416231 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 msgr2=0x7f702c13da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 --2- 172.21.15.139:0/871416231 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c13da30 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f7024009d90 tx=0x7f702402be10 comp rx=0 tx=0).stop 2024-07-26T23:59:01.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 -- 172.21.15.139:0/871416231 shutdown_connections 2024-07-26T23:59:01.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 --2- 172.21.15.139:0/871416231 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c13da30 unknown :-1 s=CLOSED pgs=3219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.981 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 --2- 172.21.15.139:0/871416231 >> v2:172.21.15.177:3300/0 conn(0x7f702c0b6560 0x7f702c0b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 -- 172.21.15.139:0/871416231 >> 172.21.15.139:0/871416231 conn(0x7f702c06e600 msgr2=0x7f702c0684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 -- 172.21.15.139:0/871416231 shutdown_connections 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.960+0000 7f7031ddf640 1 -- 172.21.15.139:0/871416231 wait complete. 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 10 monclient: build_initial_monmap 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 10 monclient: monmap: 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:59:01.982 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 1 librados: starting msgr at 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 1 librados: starting objecter 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 1 Processor -- start 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 1 -- start start 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 1 librados: setting wanted keys 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 1 librados: calling monclient init 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.961+0000 7f7031ddf640 10 monclient: init 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient: _add_conns ranks=[1,0,2] 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f702c0b6560 0x7f702c14af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): picked mon.noname-b con 0x7f702c0b6560 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:59:01.983 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f702c0b6940 0x7f702c14b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): picked mon.noname-a con 0x7f702c0b6940 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c1dc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): picked mon.noname-c con 0x7f702c13d600 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f702c14e440 con 0x7f702c0b6940 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f702c14e2c0 con 0x7f702c0b6560 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f702c14e5c0 con 0x7f702c13d600 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.962+0000 7f7031ddf640 10 monclient(hunting): authenticate will time out at 2774.998491s 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.964+0000 7f702affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f702c0b6940 0x7f702c14b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.984 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.964+0000 7f702affd640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f702c0b6940 0x7f702c14b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:51556/0 (socket says 172.21.15.139:51556) 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.964+0000 7f702affd640 1 -- 172.21.15.139:0/3177257599 learned_addr learned my addr 172.21.15.139:0/3177257599 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.964+0000 7f702affd640 10 monclient(hunting): get_auth_request con 0x7f702c0b6940 auth_method 0 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.964+0000 7f702affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.964+0000 7f702affd640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.964+0000 7f702affd640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.965+0000 7f702bfff640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c1dc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.965+0000 7f702affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.965+0000 7f702affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.966+0000 7f702affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.969+0000 7f702bfff640 10 monclient(hunting): get_auth_request con 0x7f702c13d600 auth_method 0 2024-07-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.969+0000 7f702bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.969+0000 7f702bfff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.969+0000 7f702bfff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 10 monclient(hunting): handle_auth_done global_id 15618 payload 824 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 1 -- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 msgr2=0x7f702c1dc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c1dc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 1 -- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f702c0b6560 msgr2=0x7f702c14af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f702c0b6560 0x7f702c14af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 10 monclient: _finish_hunting 0 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 1 monclient: found mon.noname-a 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.970+0000 7f702affd640 1 -- 172.21.15.139:0/3177257599 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702c14cd60 con 0x7f702c0b6940 2024-07-26T23:59:01.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.977+0000 7f702affd640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f702c0b6940 0x7f702c14b4d0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f702000c4d0 tx=0x7f702000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:01.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.977+0000 7f7013fff640 10 client.?.objecter ms_handle_connect 0x7f702c0b6940 2024-07-26T23:59:01.992 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:01.977+0000 7f7013fff640 10 client.?.objecter resend_mon_ops 2024-07-26T23:59:02.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 1 -- 172.21.15.139:0/3177257599 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020017070 con 0x7f702c0b6940 2024-07-26T23:59:02.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:02.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:59:02.039 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: dump: 2024-07-26T23:59:02.039 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:02.039 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:02.039 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: _finish_auth 0 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: _check_auth_tickets 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 1 -- 172.21.15.139:0/3177257599 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7020005d00 con 0x7f702c0b6940 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 5 monclient: authenticate success, global_id 15618 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 10 monclient: _renew_subs 2024-07-26T23:59:02.040 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f702c149b00 con 0x7f702c0b6940 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 1 -- 172.21.15.139:0/3177257599 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70200072b0 con 0x7f702c0b6940 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7013fff640 10 monclient: dump: 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:02.041 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 10 client.15618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 10 monclient: _renew_subs 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f702c1dce60 con 0x7f702c0b6940 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7031ddf640 1 librados: init done 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.023+0000 7f7029ffb640 4 set_mon_vals no callback set 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.024+0000 7f7031ddf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.024+0000 7f7031ddf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.024+0000 7f7031ddf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.139:3300/0 2024-07-26T23:59:02.042 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.024+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f702c0b4260 con 0x7f702c0b6940 2024-07-26T23:59:02.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.296+0000 7f7013fff640 1 -- 172.21.15.139:0/3177257599 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f702001a070 con 0x7f702c0b6940 2024-07-26T23:59:02.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.297+0000 7f7013fff640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f70180789f0 0x7f701807aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:02.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.297+0000 7f702b7fe640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f70180789f0 0x7f701807aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:02.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.297+0000 7f702b7fe640 10 monclient: get_auth_request con 0x7f70180789f0 auth_method 0 2024-07-26T23:59:02.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.298+0000 7f7013fff640 10 client.15618.objecter ms_handle_connect 0x7f70180789f0 2024-07-26T23:59:02.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.298+0000 7f702b7fe640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f70180789f0 0x7f701807aeb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f701c0098f0 tx=0x7f701c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:02.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 1 -- 172.21.15.139:0/3177257599 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(970..970 src has 735..970) ==== 6365+0+0 (secure 0 0 0) 0x7f702005cb90 con 0x7f702c0b6940 2024-07-26T23:59:02.326 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 10 client.15618.objecter ms_dispatch 0x7f702c000b90 osd_map(970..970 src has 735..970) 2024-07-26T23:59:02.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 3 client.15618.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:59:02.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 3 client.15618.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:59:02.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 20 client.15618.objecter dump_active .. 0 homeless 2024-07-26T23:59:02.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 1 -- 172.21.15.139:0/3177257599 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f7020021080 con 0x7f702c0b6940 2024-07-26T23:59:02.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:02.327 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.311+0000 7f7013fff640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:59:02.537 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.521+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f702c1dc460 con 0x7f70180789f0 2024-07-26T23:59:02.539 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.524+0000 7f7013fff640 1 -- 172.21.15.139:0/3177257599 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+115809 (secure 0 0 0) 0x7f701c00d7e0 con 0x7f70180789f0 2024-07-26T23:59:02.540 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:59:02.541 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 10 librados: watch_flush enter 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 10 librados: watch_flush exit 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 20 client.15618.objecter shutdown clearing up homeless session... 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 10 client.15618.objecter successfully canceled tick 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f70180789f0 msgr2=0x7f701807aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f70180789f0 0x7f701807aeb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f701c0098f0 tx=0x7f701c008040 comp rx=0 tx=0).stop 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 10 monclient: shutdown 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f702c0b6940 msgr2=0x7f702c14b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f702c0b6940 0x7f702c14b4d0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f702000c4d0 tx=0x7f702000c990 comp rx=0 tx=0).stop 2024-07-26T23:59:02.543 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 shutdown_connections 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f70180789f0 0x7f701807aeb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f702c13d600 0x7f702c1dc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f702c0b6940 0x7f702c14b4d0 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 --2- 172.21.15.139:0/3177257599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f702c0b6560 0x7f702c14af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 >> 172.21.15.139:0/3177257599 conn(0x7f702c06e600 msgr2=0x7f702c0b93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 shutdown_connections 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 -- 172.21.15.139:0/3177257599 wait complete. 2024-07-26T23:59:02.544 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:02.526+0000 7f7031ddf640 1 librados: shutdown 2024-07-26T23:59:02.554 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":1694,"stamp":"2024-07-26T23:59:02.218203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":52,"num_read_kb":43,"num_write":66,"num_write_kb":593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":711216,"kb_used_data":118432,"kb_used_omap":12,"kb_used_meta":592179,"kb_avail":754263504,"statfs":{"total":773094113280,"available":772365828096,"internally_reserved":0,"allocated":121274368,"data_stored":76592968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":606391856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002018"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1861,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.110432+0000","last_change":"2024-07-26T23:36:37.015743+0000","last_active":"2024-07-26T23:58:56.110432+0000","last_peered":"2024-07-26T23:58:56.110432+0000","last_clean":"2024-07-26T23:58:56.110432+0000","last_became_active":"2024-07-26T23:36:37.010177+0000","last_became_peered":"2024-07-26T23:36:37.010177+0000","last_unstale":"2024-07-26T23:58:56.110432+0000","last_undegraded":"2024-07-26T23:58:56.110432+0000","last_fullsized":"2024-07-26T23:58:56.110432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:54:54.240713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1872,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.924170+0000","last_change":"2024-07-26T23:36:32.386250+0000","last_active":"2024-07-26T23:59:00.924170+0000","last_peered":"2024-07-26T23:59:00.924170+0000","last_clean":"2024-07-26T23:59:00.924170+0000","last_became_active":"2024-07-26T23:36:32.380680+0000","last_became_peered":"2024-07-26T23:36:32.380680+0000","last_unstale":"2024-07-26T23:59:00.924170+0000","last_undegraded":"2024-07-26T23:59:00.924170+0000","last_fullsized":"2024-07-26T23:59:00.924170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1851,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.445927+0000","last_change":"2024-07-26T23:36:32.478927+0000","last_active":"2024-07-26T23:58:55.445927+0000","last_peered":"2024-07-26T23:58:55.445927+0000","last_clean":"2024-07-26T23:58:55.445927+0000","last_became_active":"2024-07-26T23:36:32.473374+0000","last_became_peered":"2024-07-26T23:36:32.473374+0000","last_unstale":"2024-07-26T23:58:55.445927+0000","last_undegraded":"2024-07-26T23:58:55.445927+0000","last_fullsized":"2024-07-26T23:58:55.445927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1855,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.040740+0000","last_change":"2024-07-26T23:36:45.735883+0000","last_active":"2024-07-26T23:58:56.040740+0000","last_peered":"2024-07-26T23:58:56.040740+0000","last_clean":"2024-07-26T23:58:56.040740+0000","last_became_active":"2024-07-26T23:36:45.718574+0000","last_became_peered":"2024-07-26T23:36:45.718574+0000","last_unstale":"2024-07-26T23:58:56.040740+0000","last_undegraded":"2024-07-26T23:58:56.040740+0000","last_fullsized":"2024-07-26T23:58:56.040740+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1864,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:57.256072+0000","last_change":"2024-07-26T23:58:57.256896+0000","last_active":"2024-07-26T23:58:57.256072+0000","last_peered":"2024-07-26T23:58:57.256072+0000","last_clean":"2024-07-26T23:58:57.256072+0000","last_became_active":"2024-07-26T23:36:33.915707+0000","last_became_peered":"2024-07-26T23:36:33.915707+0000","last_unstale":"2024-07-26T23:58:57.256072+0000","last_undegraded":"2024-07-26T23:58:57.256072+0000","last_fullsized":"2024-07-26T23:58:57.256072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:57.254401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:57.254401+0000","last_clean_scrub_stamp":"2024-07-26T23:58:57.254401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:32:54.370467+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"970'8","reported_seq":1871,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.123957+0000","last_change":"2024-07-26T23:58:58.124977+0000","last_active":"2024-07-26T23:58:58.123957+0000","last_peered":"2024-07-26T23:58:58.123957+0000","last_clean":"2024-07-26T23:58:58.123957+0000","last_became_active":"2024-07-26T23:36:30.507651+0000","last_became_peered":"2024-07-26T23:36:30.507651+0000","last_unstale":"2024-07-26T23:58:58.123957+0000","last_undegraded":"2024-07-26T23:58:58.123957+0000","last_fullsized":"2024-07-26T23:58:58.123957+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'8","last_scrub_stamp":"2024-07-26T23:58:58.122141+0000","last_deep_scrub":"970'8","last_deep_scrub_stamp":"2024-07-26T23:58:58.122141+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.122141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:57:29.495918+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1858,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.045416+0000","last_change":"2024-07-26T23:58:59.046267+0000","last_active":"2024-07-26T23:58:59.045416+0000","last_peered":"2024-07-26T23:58:59.045416+0000","last_clean":"2024-07-26T23:58:59.045416+0000","last_became_active":"2024-07-26T23:36:29.468516+0000","last_became_peered":"2024-07-26T23:36:29.468516+0000","last_unstale":"2024-07-26T23:58:59.045416+0000","last_undegraded":"2024-07-26T23:58:59.045416+0000","last_fullsized":"2024-07-26T23:58:59.045416+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:59.043704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:59.043704+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.043704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:12:25.111037+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1876,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.427811+0000","last_change":"2024-07-26T23:36:50.102068+0000","last_active":"2024-07-26T23:58:55.427811+0000","last_peered":"2024-07-26T23:58:55.427811+0000","last_clean":"2024-07-26T23:58:55.427811+0000","last_became_active":"2024-07-26T23:36:50.085394+0000","last_became_peered":"2024-07-26T23:36:50.085394+0000","last_unstale":"2024-07-26T23:58:55.427811+0000","last_undegraded":"2024-07-26T23:58:55.427811+0000","last_fullsized":"2024-07-26T23:58:55.427811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1858,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.080153+0000","last_change":"2024-07-26T23:58:58.081649+0000","last_active":"2024-07-26T23:58:58.080153+0000","last_peered":"2024-07-26T23:58:58.080153+0000","last_clean":"2024-07-26T23:58:58.080153+0000","last_became_active":"2024-07-26T23:36:32.315924+0000","last_became_peered":"2024-07-26T23:36:32.315924+0000","last_unstale":"2024-07-26T23:58:58.080153+0000","last_undegraded":"2024-07-26T23:58:58.080153+0000","last_fullsized":"2024-07-26T23:58:58.080153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:58.077277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:58.077277+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.077277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:45:51.170626+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1863,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.295828+0000","last_change":"2024-07-26T23:58:59.296709+0000","last_active":"2024-07-26T23:58:59.295828+0000","last_peered":"2024-07-26T23:58:59.295828+0000","last_clean":"2024-07-26T23:58:59.295828+0000","last_became_active":"2024-07-26T23:36:33.069066+0000","last_became_peered":"2024-07-26T23:36:33.069066+0000","last_unstale":"2024-07-26T23:58:59.295828+0000","last_undegraded":"2024-07-26T23:58:59.295828+0000","last_fullsized":"2024-07-26T23:58:59.295828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:59.294030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:59.294030+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.294030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:25:36.061049+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1873,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.377126+0000","last_change":"2024-07-26T23:36:32.994273+0000","last_active":"2024-07-26T23:58:55.377126+0000","last_peered":"2024-07-26T23:58:55.377126+0000","last_clean":"2024-07-26T23:58:55.377126+0000","last_became_active":"2024-07-26T23:36:32.984857+0000","last_became_peered":"2024-07-26T23:36:32.984857+0000","last_unstale":"2024-07-26T23:58:55.377126+0000","last_undegraded":"2024-07-26T23:58:55.377126+0000","last_fullsized":"2024-07-26T23:58:55.377126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1863,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.128408+0000","last_change":"2024-07-26T23:36:30.566094+0000","last_active":"2024-07-26T23:58:56.128408+0000","last_peered":"2024-07-26T23:58:56.128408+0000","last_clean":"2024-07-26T23:58:56.128408+0000","last_became_active":"2024-07-26T23:36:30.560131+0000","last_became_peered":"2024-07-26T23:36:30.560131+0000","last_unstale":"2024-07-26T23:58:56.128408+0000","last_undegraded":"2024-07-26T23:58:56.128408+0000","last_fullsized":"2024-07-26T23:58:56.128408+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:58:36.316935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1856,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.437764+0000","last_change":"2024-07-26T23:36:44.212854+0000","last_active":"2024-07-26T23:58:58.437764+0000","last_peered":"2024-07-26T23:58:58.437764+0000","last_clean":"2024-07-26T23:58:58.437764+0000","last_became_active":"2024-07-26T23:36:44.149525+0000","last_became_peered":"2024-07-26T23:36:44.149525+0000","last_unstale":"2024-07-26T23:58:58.437764+0000","last_undegraded":"2024-07-26T23:58:58.437764+0000","last_fullsized":"2024-07-26T23:58:58.437764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"970'34","reported_seq":1884,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:57.185708+0000","last_change":"2024-07-26T23:58:57.187263+0000","last_active":"2024-07-26T23:58:57.185708+0000","last_peered":"2024-07-26T23:58:57.185708+0000","last_clean":"2024-07-26T23:58:57.185708+0000","last_became_active":"2024-07-26T23:35:16.707863+0000","last_became_peered":"2024-07-26T23:35:16.707863+0000","last_unstale":"2024-07-26T23:58:57.185708+0000","last_undegraded":"2024-07-26T23:58:57.185708+0000","last_fullsized":"2024-07-26T23:58:57.185708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'34","last_scrub_stamp":"2024-07-26T23:58:57.182818+0000","last_deep_scrub":"970'34","last_deep_scrub_stamp":"2024-07-26T23:58:57.182818+0000","last_clean_scrub_stamp":"2024-07-26T23:58:57.182818+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:09:47.734483+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"970'2","reported_seq":1891,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.566927+0000","last_change":"2024-07-26T23:59:00.567946+0000","last_active":"2024-07-26T23:59:00.566927+0000","last_peered":"2024-07-26T23:59:00.566927+0000","last_clean":"2024-07-26T23:59:00.566927+0000","last_became_active":"2024-07-26T23:35:19.087174+0000","last_became_peered":"2024-07-26T23:35:19.087174+0000","last_unstale":"2024-07-26T23:59:00.566927+0000","last_undegraded":"2024-07-26T23:59:00.566927+0000","last_fullsized":"2024-07-26T23:59:00.566927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'2","last_scrub_stamp":"2024-07-26T23:59:00.565123+0000","last_deep_scrub":"970'2","last_deep_scrub_stamp":"2024-07-26T23:59:00.565123+0000","last_clean_scrub_stamp":"2024-07-26T23:59:00.565123+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:23:44.786699+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0.074002362000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1862,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.123049+0000","last_change":"2024-07-26T23:36:39.448493+0000","last_active":"2024-07-26T23:58:56.123049+0000","last_peered":"2024-07-26T23:58:56.123049+0000","last_clean":"2024-07-26T23:58:56.123049+0000","last_became_active":"2024-07-26T23:36:39.441853+0000","last_became_peered":"2024-07-26T23:36:39.441853+0000","last_unstale":"2024-07-26T23:58:56.123049+0000","last_undegraded":"2024-07-26T23:58:56.123049+0000","last_fullsized":"2024-07-26T23:58:56.123049+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:33:54.532094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1889,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.538091+0000","last_change":"2024-07-26T23:58:58.539215+0000","last_active":"2024-07-26T23:58:58.538091+0000","last_peered":"2024-07-26T23:58:58.538091+0000","last_clean":"2024-07-26T23:58:58.538091+0000","last_became_active":"2024-07-26T23:35:17.846505+0000","last_became_peered":"2024-07-26T23:35:17.846505+0000","last_unstale":"2024-07-26T23:58:58.538091+0000","last_undegraded":"2024-07-26T23:58:58.538091+0000","last_fullsized":"2024-07-26T23:58:58.538091+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:58.535053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:58.535053+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.535053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:14:14.627697+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0.11625252899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1853,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.044413+0000","last_change":"2024-07-26T23:36:33.996623+0000","last_active":"2024-07-26T23:58:56.044413+0000","last_peered":"2024-07-26T23:58:56.044413+0000","last_clean":"2024-07-26T23:58:56.044413+0000","last_became_active":"2024-07-26T23:36:33.958339+0000","last_became_peered":"2024-07-26T23:36:33.958339+0000","last_unstale":"2024-07-26T23:58:56.044413+0000","last_undegraded":"2024-07-26T23:58:56.044413+0000","last_fullsized":"2024-07-26T23:58:56.044413+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1887,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.108631+0000","last_change":"2024-07-26T23:35:22.093863+0000","last_active":"2024-07-26T23:58:56.108631+0000","last_peered":"2024-07-26T23:58:56.108631+0000","last_clean":"2024-07-26T23:58:56.108631+0000","last_became_active":"2024-07-26T23:35:17.814359+0000","last_became_peered":"2024-07-26T23:35:17.814359+0000","last_unstale":"2024-07-26T23:58:56.108631+0000","last_undegraded":"2024-07-26T23:58:56.108631+0000","last_fullsized":"2024-07-26T23:58:56.108631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:27:36.663020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060284956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1864,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.119056+0000","last_change":"2024-07-26T23:36:32.118711+0000","last_active":"2024-07-26T23:58:56.119056+0000","last_peered":"2024-07-26T23:58:56.119056+0000","last_clean":"2024-07-26T23:58:56.119056+0000","last_became_active":"2024-07-26T23:36:32.112540+0000","last_became_peered":"2024-07-26T23:36:32.112540+0000","last_unstale":"2024-07-26T23:58:56.119056+0000","last_undegraded":"2024-07-26T23:58:56.119056+0000","last_fullsized":"2024-07-26T23:58:56.119056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:32:07.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"970'3","reported_seq":1892,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.557334+0000","last_change":"2024-07-26T23:58:59.558707+0000","last_active":"2024-07-26T23:58:59.557334+0000","last_peered":"2024-07-26T23:58:59.557334+0000","last_clean":"2024-07-26T23:58:59.557334+0000","last_became_active":"2024-07-26T23:35:18.492082+0000","last_became_peered":"2024-07-26T23:35:18.492082+0000","last_unstale":"2024-07-26T23:58:59.557334+0000","last_undegraded":"2024-07-26T23:58:59.557334+0000","last_fullsized":"2024-07-26T23:58:59.557334+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'3","last_scrub_stamp":"2024-07-26T23:58:59.555471+0000","last_deep_scrub":"970'3","last_deep_scrub_stamp":"2024-07-26T23:58:59.555471+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.555471+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:40:19.263535+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0.098681633000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1876,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.423231+0000","last_change":"2024-07-26T23:36:29.379730+0000","last_active":"2024-07-26T23:58:55.423231+0000","last_peered":"2024-07-26T23:58:55.423231+0000","last_clean":"2024-07-26T23:58:55.423231+0000","last_became_active":"2024-07-26T23:36:29.374355+0000","last_became_peered":"2024-07-26T23:36:29.374355+0000","last_unstale":"2024-07-26T23:58:55.423231+0000","last_undegraded":"2024-07-26T23:58:55.423231+0000","last_fullsized":"2024-07-26T23:58:55.423231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1885,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.112470+0000","last_change":"2024-07-26T23:35:22.376891+0000","last_active":"2024-07-26T23:58:56.112470+0000","last_peered":"2024-07-26T23:58:56.112470+0000","last_clean":"2024-07-26T23:58:56.112470+0000","last_became_active":"2024-07-26T23:35:18.507138+0000","last_became_peered":"2024-07-26T23:35:18.507138+0000","last_unstale":"2024-07-26T23:58:56.112470+0000","last_undegraded":"2024-07-26T23:58:56.112470+0000","last_fullsized":"2024-07-26T23:58:56.112470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:56:22.382897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047472209000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1864,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.110576+0000","last_change":"2024-07-26T23:36:32.850160+0000","last_active":"2024-07-26T23:58:56.110576+0000","last_peered":"2024-07-26T23:58:56.110576+0000","last_clean":"2024-07-26T23:58:56.110576+0000","last_became_active":"2024-07-26T23:36:32.842079+0000","last_became_peered":"2024-07-26T23:36:32.842079+0000","last_unstale":"2024-07-26T23:58:56.110576+0000","last_undegraded":"2024-07-26T23:58:56.110576+0000","last_fullsized":"2024-07-26T23:58:56.110576+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:53:03.317423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1880,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.026837+0000","last_change":"2024-07-26T23:35:22.251048+0000","last_active":"2024-07-26T23:58:56.026837+0000","last_peered":"2024-07-26T23:58:56.026837+0000","last_clean":"2024-07-26T23:58:56.026837+0000","last_became_active":"2024-07-26T23:35:17.925979+0000","last_became_peered":"2024-07-26T23:35:17.925979+0000","last_unstale":"2024-07-26T23:58:56.026837+0000","last_undegraded":"2024-07-26T23:58:56.026837+0000","last_fullsized":"2024-07-26T23:58:56.026837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073846415999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1864,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.096227+0000","last_change":"2024-07-26T23:36:31.631363+0000","last_active":"2024-07-26T23:58:56.096227+0000","last_peered":"2024-07-26T23:58:56.096227+0000","last_clean":"2024-07-26T23:58:56.096227+0000","last_became_active":"2024-07-26T23:36:31.543107+0000","last_became_peered":"2024-07-26T23:36:31.543107+0000","last_unstale":"2024-07-26T23:58:56.096227+0000","last_undegraded":"2024-07-26T23:58:56.096227+0000","last_fullsized":"2024-07-26T23:58:56.096227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:49:57.654672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1905,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.175639+0000","last_change":"2024-07-26T23:59:00.176501+0000","last_active":"2024-07-26T23:59:00.175639+0000","last_peered":"2024-07-26T23:59:00.175639+0000","last_clean":"2024-07-26T23:59:00.175639+0000","last_became_active":"2024-07-26T23:35:18.587303+0000","last_became_peered":"2024-07-26T23:35:18.587303+0000","last_unstale":"2024-07-26T23:59:00.175639+0000","last_undegraded":"2024-07-26T23:59:00.175639+0000","last_fullsized":"2024-07-26T23:59:00.175639+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:00.174009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:00.174009+0000","last_clean_scrub_stamp":"2024-07-26T23:59:00.174009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:36:10.433424+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.077704690000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1873,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.363871+0000","last_change":"2024-07-26T23:36:50.276152+0000","last_active":"2024-07-26T23:58:55.363871+0000","last_peered":"2024-07-26T23:58:55.363871+0000","last_clean":"2024-07-26T23:58:55.363871+0000","last_became_active":"2024-07-26T23:36:50.258118+0000","last_became_peered":"2024-07-26T23:36:50.258118+0000","last_unstale":"2024-07-26T23:58:55.363871+0000","last_undegraded":"2024-07-26T23:58:55.363871+0000","last_fullsized":"2024-07-26T23:58:55.363871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1877,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.042586+0000","last_change":"2024-07-26T23:35:22.236663+0000","last_active":"2024-07-26T23:58:56.042586+0000","last_peered":"2024-07-26T23:58:56.042586+0000","last_clean":"2024-07-26T23:58:56.042586+0000","last_became_active":"2024-07-26T23:35:18.556903+0000","last_became_peered":"2024-07-26T23:35:18.556903+0000","last_unstale":"2024-07-26T23:58:56.042586+0000","last_undegraded":"2024-07-26T23:58:56.042586+0000","last_fullsized":"2024-07-26T23:58:56.042586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:35:16.227696+0000","last_clean_scrub_stamp":"2024-07-26T23:35:16.227696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064553026999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1862,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.323480+0000","last_change":"2024-07-26T23:58:58.324317+0000","last_active":"2024-07-26T23:58:58.323480+0000","last_peered":"2024-07-26T23:58:58.323480+0000","last_clean":"2024-07-26T23:58:58.323480+0000","last_became_active":"2024-07-26T23:36:28.821361+0000","last_became_peered":"2024-07-26T23:36:28.821361+0000","last_unstale":"2024-07-26T23:58:58.323480+0000","last_undegraded":"2024-07-26T23:58:58.323480+0000","last_fullsized":"2024-07-26T23:58:58.323480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:58.321762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:58.321762+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.321762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:10:03.939499+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1849,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.424219+0000","last_change":"2024-07-26T23:36:29.506650+0000","last_active":"2024-07-26T23:58:55.424219+0000","last_peered":"2024-07-26T23:58:55.424219+0000","last_clean":"2024-07-26T23:58:55.424219+0000","last_became_active":"2024-07-26T23:36:29.496365+0000","last_became_peered":"2024-07-26T23:36:29.496365+0000","last_unstale":"2024-07-26T23:58:55.424219+0000","last_undegraded":"2024-07-26T23:58:55.424219+0000","last_fullsized":"2024-07-26T23:58:55.424219+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1863,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.106726+0000","last_change":"2024-07-26T23:36:33.306681+0000","last_active":"2024-07-26T23:58:56.106726+0000","last_peered":"2024-07-26T23:58:56.106726+0000","last_clean":"2024-07-26T23:58:56.106726+0000","last_became_active":"2024-07-26T23:36:33.301227+0000","last_became_peered":"2024-07-26T23:36:33.301227+0000","last_unstale":"2024-07-26T23:58:56.106726+0000","last_undegraded":"2024-07-26T23:58:56.106726+0000","last_fullsized":"2024-07-26T23:58:56.106726+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:47:53.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1863,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.098148+0000","last_change":"2024-07-26T23:36:30.287713+0000","last_active":"2024-07-26T23:58:56.098148+0000","last_peered":"2024-07-26T23:58:56.098148+0000","last_clean":"2024-07-26T23:58:56.098148+0000","last_became_active":"2024-07-26T23:36:30.280064+0000","last_became_peered":"2024-07-26T23:36:30.280064+0000","last_unstale":"2024-07-26T23:58:56.098148+0000","last_undegraded":"2024-07-26T23:58:56.098148+0000","last_fullsized":"2024-07-26T23:58:56.098148+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:43:33.794147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1846,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.380028+0000","last_change":"2024-07-26T23:36:29.655104+0000","last_active":"2024-07-26T23:58:55.380028+0000","last_peered":"2024-07-26T23:58:55.380028+0000","last_clean":"2024-07-26T23:58:55.380028+0000","last_became_active":"2024-07-26T23:36:29.640586+0000","last_became_peered":"2024-07-26T23:36:29.640586+0000","last_unstale":"2024-07-26T23:58:55.380028+0000","last_undegraded":"2024-07-26T23:58:55.380028+0000","last_fullsized":"2024-07-26T23:58:55.380028+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1849,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.447773+0000","last_change":"2024-07-26T23:36:28.901469+0000","last_active":"2024-07-26T23:58:55.447773+0000","last_peered":"2024-07-26T23:58:55.447773+0000","last_clean":"2024-07-26T23:58:55.447773+0000","last_became_active":"2024-07-26T23:36:28.852843+0000","last_became_peered":"2024-07-26T23:36:28.852843+0000","last_unstale":"2024-07-26T23:58:55.447773+0000","last_undegraded":"2024-07-26T23:58:55.447773+0000","last_fullsized":"2024-07-26T23:58:55.447773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1855,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.085677+0000","last_change":"2024-07-26T23:58:59.086559+0000","last_active":"2024-07-26T23:58:59.085677+0000","last_peered":"2024-07-26T23:58:59.085677+0000","last_clean":"2024-07-26T23:58:59.085677+0000","last_became_active":"2024-07-26T23:36:31.124146+0000","last_became_peered":"2024-07-26T23:36:31.124146+0000","last_unstale":"2024-07-26T23:58:59.085677+0000","last_undegraded":"2024-07-26T23:58:59.085677+0000","last_fullsized":"2024-07-26T23:58:59.085677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:59.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:59.082134+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:08:17.106823+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1862,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.252186+0000","last_change":"2024-07-26T23:58:56.253051+0000","last_active":"2024-07-26T23:58:56.252186+0000","last_peered":"2024-07-26T23:58:56.252186+0000","last_clean":"2024-07-26T23:58:56.252186+0000","last_became_active":"2024-07-26T23:36:30.173625+0000","last_became_peered":"2024-07-26T23:36:30.173625+0000","last_unstale":"2024-07-26T23:58:56.252186+0000","last_undegraded":"2024-07-26T23:58:56.252186+0000","last_fullsized":"2024-07-26T23:58:56.252186+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:56.250348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:56.250348+0000","last_clean_scrub_stamp":"2024-07-26T23:58:56.250348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:18:56.590554+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1861,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.106709+0000","last_change":"2024-07-26T23:36:31.996757+0000","last_active":"2024-07-26T23:58:56.106709+0000","last_peered":"2024-07-26T23:58:56.106709+0000","last_clean":"2024-07-26T23:58:56.106709+0000","last_became_active":"2024-07-26T23:36:31.990984+0000","last_became_peered":"2024-07-26T23:36:31.990984+0000","last_unstale":"2024-07-26T23:58:56.106709+0000","last_undegraded":"2024-07-26T23:58:56.106709+0000","last_fullsized":"2024-07-26T23:58:56.106709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:24:44.872192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1875,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.421364+0000","last_change":"2024-07-26T23:36:33.389570+0000","last_active":"2024-07-26T23:58:55.421364+0000","last_peered":"2024-07-26T23:58:55.421364+0000","last_clean":"2024-07-26T23:58:55.421364+0000","last_became_active":"2024-07-26T23:36:33.382005+0000","last_became_peered":"2024-07-26T23:36:33.382005+0000","last_unstale":"2024-07-26T23:58:55.421364+0000","last_undegraded":"2024-07-26T23:58:55.421364+0000","last_fullsized":"2024-07-26T23:58:55.421364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1849,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.420230+0000","last_change":"2024-07-26T23:36:28.719222+0000","last_active":"2024-07-26T23:58:55.420230+0000","last_peered":"2024-07-26T23:58:55.420230+0000","last_clean":"2024-07-26T23:58:55.420230+0000","last_became_active":"2024-07-26T23:36:28.654711+0000","last_became_peered":"2024-07-26T23:36:28.654711+0000","last_unstale":"2024-07-26T23:58:55.420230+0000","last_undegraded":"2024-07-26T23:58:55.420230+0000","last_fullsized":"2024-07-26T23:58:55.420230+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1872,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:55.356368+0000","last_change":"2024-07-26T23:36:36.812820+0000","last_active":"2024-07-26T23:58:55.356368+0000","last_peered":"2024-07-26T23:58:55.356368+0000","last_clean":"2024-07-26T23:58:55.356368+0000","last_became_active":"2024-07-26T23:36:36.803418+0000","last_became_peered":"2024-07-26T23:36:36.803418+0000","last_unstale":"2024-07-26T23:58:55.356368+0000","last_undegraded":"2024-07-26T23:58:55.356368+0000","last_fullsized":"2024-07-26T23:58:55.356368+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:36:23.254869+0000","last_clean_scrub_stamp":"2024-07-26T23:36:23.254869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":21,"seq":90194313505,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94456,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94277384,"statfs":{"total":96636764160,"available":96540041216,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313509,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90296,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281544,"statfs":{"total":96636764160,"available":96544301056,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":20,"seq":85899346210,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90300,"kb_used_data":14708,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281540,"statfs":{"total":96636764160,"available":96544296960,"internally_reserved":0,"allocated":15060992,"data_stored":9474674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346213,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90692,"kb_used_data":15100,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281148,"statfs":{"total":96636764160,"available":96543895552,"internally_reserved":0,"allocated":15462400,"data_stored":9872500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411618,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82152,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94289688,"statfs":{"total":96636764160,"available":96552640512,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509730,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90472,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281368,"statfs":{"total":96636764160,"available":96544120832,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575137,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77824,"kb_used_data":14712,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94294016,"statfs":{"total":96636764160,"available":96557072384,"internally_reserved":0,"allocated":15065088,"data_stored":9474679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575135,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95024,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94276816,"statfs":{"total":96636764160,"available":96539459584,"internally_reserved":0,"allocated":15458304,"data_stored":9872495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T23:59:02.560 INFO:tasks.ceph:pgid 12.1c last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.560 INFO:tasks.ceph:pgid 12.1d last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.560 INFO:tasks.ceph:pgid 12.1e last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.560 INFO:tasks.ceph:pgid 12.1f last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.560 INFO:tasks.ceph:pgid 12.3 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.561 INFO:tasks.ceph:pgid 12.6 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.561 INFO:tasks.ceph:pgid 12.7 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.561 INFO:tasks.ceph:pgid 12.d last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.561 INFO:tasks.ceph:pgid 12.e last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.561 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.561 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-07-26T23:35:16.227696+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=35, tm_sec=16, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.561 INFO:tasks.ceph:pgid 12.c last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.562 INFO:tasks.ceph:pgid 12.a last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.562 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-07-26T23:35:16.227696+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=35, tm_sec=16, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.562 INFO:tasks.ceph:pgid 12.b last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.562 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-07-26T23:35:16.227696+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=35, tm_sec=16, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.562 INFO:tasks.ceph:pgid 12.8 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.562 INFO:tasks.ceph:pgid 12.9 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.562 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-07-26T23:35:16.227696+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=35, tm_sec=16, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.1a last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.19 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.18 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.17 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.16 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.13 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.12 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.11 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.563 INFO:tasks.ceph:pgid 12.10 last_scrub_stamp 2024-07-26T23:36:23.254869+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=36, tm_sec=23, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=23, tm_min=58, tm_sec=53, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T23:59:02.564 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-07-26T23:59:05.536 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:13.194 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.540 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:13.198 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.552 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:13.209 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.588 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:13.246 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.591 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:13.249 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.593 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:13.242 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.594 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:13.243 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.604 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:13.262 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:05.606 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:13.255 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.578 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:28.235 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.579 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:28.236 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.593 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:28.250 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.628 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:28.276 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.631 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:28.280 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.634 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:28.292 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.638 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:28.296 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.649 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:28.297 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:20.654 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:28.311 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.351 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:29.008 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.351 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:29.000 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.352 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:29.009 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.352 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:29.001 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.361 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:29.018 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.361 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:29.010 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.385 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:29.043 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.386 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:29.044 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:21.395 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:29.053 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:22.565 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T23:59:22.667 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.651+0000 7fe843f70640 10 monclient: get_monmap_and_config 2024-07-26T23:59:22.667 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.651+0000 7fe843f70640 10 monclient: build_initial_monmap 2024-07-26T23:59:22.667 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.651+0000 7fe843f70640 10 monclient: monmap: 2024-07-26T23:59:22.667 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:59:22.653008+0000 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:59:22.653008+0000 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:59:22.668 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:22.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.652+0000 7fe843f70640 1 Processor -- start 2024-07-26T23:59:22.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.652+0000 7fe843f70640 1 -- start start 2024-07-26T23:59:22.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.652+0000 7fe843f70640 10 monclient: init 2024-07-26T23:59:22.670 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fe83c0b2660 0x7fe83c074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient(hunting): picked mon.noname-a con 0x7fe83c0b2660 addr v2:172.21.15.139:3300/0 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe83c0b2a40 addr v1:172.21.15.177:6789/0 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe83c0748b0 addr v1:172.21.15.139:6789/0 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 1 -- --> v1:172.21.15.139:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83c0af400 con 0x7fe83c0748b0 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83c0af6e0 con 0x7fe83c0b2a40 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 1 -- --> v2:172.21.15.139:3300/0 -- mon_getmap magic: 0 -- 0x7fe83c0ad560 con 0x7fe83c0b2660 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.653+0000 7fe843f70640 10 monclient(hunting): authenticate will time out at 2795.689533s 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.654+0000 7fe841d0c640 1 --2- >> v2:172.21.15.139:3300/0 conn(0x7fe83c0b2660 0x7fe83c074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:22.671 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.654+0000 7fe84150b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe83c0b2a40 0x7fe83c143c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.139:36828/0 (socket says 172.21.15.139:36828) 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.654+0000 7fe84150b640 1 -- 172.21.15.139:0/478113738 learned_addr learned my addr 172.21.15.139:0/478113738 (peer_addr_for_me v1:172.21.15.139:0/0) 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.654+0000 7fe841d0c640 10 monclient(hunting): get_auth_request con 0x7fe83c0b2660 auth_method 0 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.654+0000 7fe841d0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.654+0000 7fe841d0c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.654+0000 7fe841d0c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.655+0000 7fe841d0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.655+0000 7fe841d0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:22.672 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.655+0000 7fe841d0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:22.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.656+0000 7fe840d0a640 1 -- 172.21.15.139:0/478113738 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490890794 0 0) 0x7fe83c0af6e0 con 0x7fe83c0b2a40 2024-07-26T23:59:22.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.656+0000 7fe840d0a640 10 monclient(hunting): _negotiate 2024-07-26T23:59:22.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.656+0000 7fe840d0a640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:22.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.656+0000 7fe840d0a640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:22.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.656+0000 7fe840d0a640 10 monclient(hunting): my global_id is 20365 2024-07-26T23:59:22.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.656+0000 7fe840d0a640 1 -- 172.21.15.139:0/478113738 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe828003520 con 0x7fe83c0b2a40 2024-07-26T23:59:22.673 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 10 monclient(hunting): handle_auth_done global_id 15621 payload 274 2024-07-26T23:59:22.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 1 -- 172.21.15.139:0/478113738 >> v1:172.21.15.177:6789/0 conn(0x7fe83c0b2a40 legacy=0x7fe83c143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:22.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 1 -- 172.21.15.139:0/478113738 >> v1:172.21.15.139:6789/0 conn(0x7fe83c0748b0 legacy=0x7fe83c14aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:22.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 10 monclient: _finish_hunting 0 2024-07-26T23:59:22.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 1 monclient: found mon.noname-a 2024-07-26T23:59:22.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:22.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.139:3300/0 2024-07-26T23:59:22.674 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.657+0000 7fe841d0c640 1 -- 172.21.15.139:0/478113738 --> v2:172.21.15.139:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c1447a0 con 0x7fe83c0b2660 2024-07-26T23:59:22.677 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.661+0000 7fe841d0c640 1 --2- 172.21.15.139:0/478113738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c0b2660 0x7fe83c074370 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fe83c1449d0 tx=0x7fe83802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dad5422c6289b36 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:22.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 1 -- 172.21.15.139:0/478113738 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8380315a0 con 0x7fe83c0b2660 2024-07-26T23:59:22.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:22.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-07-26T23:59:22.681 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: dump: 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:22.682 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: _finish_auth 0 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: _check_auth_tickets 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 1 -- 172.21.15.139:0/478113738 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe838037070 con 0x7fe83c0b2660 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe843f70640 5 monclient: authenticate success, global_id 15621 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe843f70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe843f70640 10 monclient: get_monmap_and_config success 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 1 -- 172.21.15.139:0/478113738 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838031960 con 0x7fe83c0b2660 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe843f70640 4 set_mon_vals no callback set 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:22.683 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.665+0000 7fe840d0a640 10 monclient: dump: 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:22.684 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:22.685 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:22.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 10 monclient: shutdown 2024-07-26T23:59:22.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:22.688 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 1 -- 172.21.15.139:0/478113738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c0b2660 msgr2=0x7fe83c074370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 1 --2- 172.21.15.139:0/478113738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c0b2660 0x7fe83c074370 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fe83c1449d0 tx=0x7fe83802b840 comp rx=0 tx=0).stop 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 1 -- 172.21.15.139:0/478113738 shutdown_connections 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 1 --2- 172.21.15.139:0/478113738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c0b2660 0x7fe83c074370 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 1 -- 172.21.15.139:0/478113738 >> 172.21.15.139:0/478113738 conn(0x7fe83c0b5b40 msgr2=0x7fe83c0b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 1 -- 172.21.15.139:0/478113738 shutdown_connections 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.666+0000 7fe843f70640 1 -- 172.21.15.139:0/478113738 wait complete. 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 10 monclient: build_initial_monmap 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 10 monclient: monmap: 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:59:22.689 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 1 librados: starting msgr at 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 1 librados: starting objecter 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 1 Processor -- start 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 1 -- start start 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 1 librados: setting wanted keys 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 1 librados: calling monclient init 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.667+0000 7fe843f70640 10 monclient: init 2024-07-26T23:59:22.690 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient: _add_conns ranks=[1,2,0] 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe83c0748b0 0x7fe83c1da640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): picked mon.noname-b con 0x7fe83c0748b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe83c0b2660 0x7fe83c136610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): picked mon.noname-c con 0x7fe83c0b2660 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c136b50 0x7fe83c1ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): picked mon.noname-a con 0x7fe83c136b50 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c0ae3a0 con 0x7fe83c136b50 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c0ae220 con 0x7fe83c0748b0 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:22.691 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7fe83c144370 con 0x7fe83c0b2660 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.668+0000 7fe843f70640 10 monclient(hunting): authenticate will time out at 2795.704600s 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84250d640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c136b50 0x7fe83c1ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe83c0748b0 0x7fe83c1da640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84150b640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe83c0b2660 0x7fe83c136610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe83c0748b0 0x7fe83c1da640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:58396/0 (socket says 172.21.15.139:58396) 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 1 -- 172.21.15.139:0/3202560580 learned_addr learned my addr 172.21.15.139:0/3202560580 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:22.692 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 10 monclient(hunting): get_auth_request con 0x7fe83c0748b0 auth_method 0 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84250d640 10 monclient(hunting): get_auth_request con 0x7fe83c136b50 auth_method 0 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84250d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84250d640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84250d640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84150b640 10 monclient(hunting): get_auth_request con 0x7fe83c0b2660 auth_method 0 2024-07-26T23:59:22.693 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84150b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84150b640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84150b640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe841d0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84250d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.669+0000 7fe84250d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:22.694 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.670+0000 7fe84250d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.670+0000 7fe84150b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.670+0000 7fe84150b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.670+0000 7fe84150b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 10 monclient(hunting): handle_auth_done global_id 20368 payload 824 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 1 -- 172.21.15.139:0/3202560580 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe83c0b2660 msgr2=0x7fe83c136610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe83c0b2660 0x7fe83c136610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 1 -- 172.21.15.139:0/3202560580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c136b50 msgr2=0x7fe83c1ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c136b50 0x7fe83c1ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 10 monclient: _finish_hunting 0 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 1 monclient: found mon.noname-b 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:59:22.695 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.672+0000 7fe841d0c640 1 -- 172.21.15.139:0/3202560580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c1cc7c0 con 0x7fe83c0748b0 2024-07-26T23:59:22.697 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.674+0000 7fe841d0c640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe83c0748b0 0x7fe83c1da640 secure :-1 s=READY pgs=4955 cs=0 l=1 rev1=1 crypto rx=0x7fe838000c00 tx=0x7fe8380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:22.697 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.674+0000 7fe8327fc640 10 client.?.objecter ms_handle_connect 0x7fe83c0748b0 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.674+0000 7fe8327fc640 10 client.?.objecter resend_mon_ops 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 1 -- 172.21.15.139:0/3202560580 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838032840 con 0x7fe83c0748b0 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: dump: 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.698 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: _finish_auth 0 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: _check_auth_tickets 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 1 -- 172.21.15.139:0/3202560580 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8380329e0 con 0x7fe83c0748b0 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 5 monclient: authenticate success, global_id 20368 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 10 monclient: _renew_subs 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83c1ccaa0 con 0x7fe83c0748b0 2024-07-26T23:59:22.699 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 1 -- 172.21.15.139:0/3202560580 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838032d90 con 0x7fe83c0748b0 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe8327fc640 10 monclient: dump: 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:22.700 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe833fff640 4 set_mon_vals no callback set 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 10 client.20368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 10 monclient: _renew_subs 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83c137000 con 0x7fe83c0748b0 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.678+0000 7fe843f70640 1 librados: init done 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.679+0000 7fe843f70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.679+0000 7fe843f70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:22.701 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.679+0000 7fe843f70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:22.702 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.679+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83c074370 con 0x7fe83c0748b0 2024-07-26T23:59:22.714 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.698+0000 7fe8327fc640 1 -- 172.21.15.139:0/3202560580 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7fe8380453f0 con 0x7fe83c0748b0 2024-07-26T23:59:22.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.699+0000 7fe8327fc640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe8200789f0 0x7fe82007aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:22.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.699+0000 7fe84150b640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe8200789f0 0x7fe82007aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:22.715 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.699+0000 7fe84150b640 10 monclient: get_auth_request con 0x7fe8200789f0 auth_method 0 2024-07-26T23:59:22.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.700+0000 7fe84150b640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe8200789f0 0x7fe82007aeb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe83c1cba60 tx=0x7fe82c007cd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:22.716 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.700+0000 7fe8327fc640 10 client.20368.objecter ms_handle_connect 0x7fe8200789f0 2024-07-26T23:59:22.717 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.702+0000 7fe8327fc640 1 -- 172.21.15.139:0/3202560580 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 735..970) ==== 6365+0+0 (secure 0 0 0) 0x7fe83802f030 con 0x7fe83c0748b0 2024-07-26T23:59:22.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.702+0000 7fe8327fc640 10 client.20368.objecter ms_dispatch 0x7fe83c000b90 osd_map(970..970 src has 735..970) 2024-07-26T23:59:22.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.702+0000 7fe8327fc640 3 client.20368.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:59:22.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.702+0000 7fe8327fc640 3 client.20368.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:59:22.718 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.702+0000 7fe8327fc640 20 client.20368.objecter dump_active .. 0 homeless 2024-07-26T23:59:22.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.936+0000 7fe8327fc640 1 -- 172.21.15.139:0/3202560580 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7fe838040070 con 0x7fe83c0748b0 2024-07-26T23:59:22.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.936+0000 7fe8327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:22.952 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:22.936+0000 7fe8327fc640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:59:23.134 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.118+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 --> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe83c13ec00 con 0x7fe8200789f0 2024-07-26T23:59:23.136 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.120+0000 7fe8327fc640 1 -- 172.21.15.139:0/3202560580 <== mgr.4101 v2:172.21.15.177:6800/1751160634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116342 (secure 0 0 0) 0x7fe83c13ec00 con 0x7fe8200789f0 2024-07-26T23:59:23.137 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-07-26T23:59:23.138 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 10 librados: watch_flush enter 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 10 librados: watch_flush exit 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 20 client.20368.objecter shutdown clearing up homeless session... 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 10 client.20368.objecter successfully canceled tick 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe8200789f0 msgr2=0x7fe82007aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe8200789f0 0x7fe82007aeb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe83c1cba60 tx=0x7fe82c007cd0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 10 monclient: shutdown 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe83c0748b0 msgr2=0x7fe83c1da640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe83c0748b0 0x7fe83c1da640 secure :-1 s=READY pgs=4955 cs=0 l=1 rev1=1 crypto rx=0x7fe838000c00 tx=0x7fe8380046c0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.140 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 shutdown_connections 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7fe8200789f0 0x7fe82007aeb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7fe83c0b2660 0x7fe83c136610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe83c136b50 0x7fe83c1ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 --2- 172.21.15.139:0/3202560580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe83c0748b0 0x7fe83c1da640 unknown :-1 s=CLOSED pgs=4955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 >> 172.21.15.139:0/3202560580 conn(0x7fe83c0b5b40 msgr2=0x7fe83c140dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 shutdown_connections 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 -- 172.21.15.139:0/3202560580 wait complete. 2024-07-26T23:59:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.123+0000 7fe843f70640 1 librados: shutdown 2024-07-26T23:59:23.149 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":1704,"stamp":"2024-07-26T23:59:22.222409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":52,"num_read_kb":43,"num_write":66,"num_write_kb":593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":710688,"kb_used_data":118432,"kb_used_omap":12,"kb_used_meta":592179,"kb_avail":754264032,"statfs":{"total":773094113280,"available":772366368768,"internally_reserved":0,"allocated":121274368,"data_stored":76592968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":606391856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1870,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:02.879235+0000","last_change":"2024-07-26T23:59:02.880063+0000","last_active":"2024-07-26T23:59:02.879235+0000","last_peered":"2024-07-26T23:59:02.879235+0000","last_clean":"2024-07-26T23:59:02.879235+0000","last_became_active":"2024-07-26T23:36:37.010177+0000","last_became_peered":"2024-07-26T23:36:37.010177+0000","last_unstale":"2024-07-26T23:59:02.879235+0000","last_undegraded":"2024-07-26T23:59:02.879235+0000","last_fullsized":"2024-07-26T23:59:02.879235+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:02.877617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:02.877617+0000","last_clean_scrub_stamp":"2024-07-26T23:59:02.877617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:02:10.037834+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1880,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:04.763628+0000","last_change":"2024-07-26T23:59:04.764461+0000","last_active":"2024-07-26T23:59:04.763628+0000","last_peered":"2024-07-26T23:59:04.763628+0000","last_clean":"2024-07-26T23:59:04.763628+0000","last_became_active":"2024-07-26T23:36:32.380680+0000","last_became_peered":"2024-07-26T23:36:32.380680+0000","last_unstale":"2024-07-26T23:59:04.763628+0000","last_undegraded":"2024-07-26T23:59:04.763628+0000","last_fullsized":"2024-07-26T23:59:04.763628+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:04.761933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:04.761933+0000","last_clean_scrub_stamp":"2024-07-26T23:59:04.761933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:39.942659+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1859,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:03.089744+0000","last_change":"2024-07-26T23:59:03.090589+0000","last_active":"2024-07-26T23:59:03.089744+0000","last_peered":"2024-07-26T23:59:03.089744+0000","last_clean":"2024-07-26T23:59:03.089744+0000","last_became_active":"2024-07-26T23:36:32.473374+0000","last_became_peered":"2024-07-26T23:36:32.473374+0000","last_unstale":"2024-07-26T23:59:03.089744+0000","last_undegraded":"2024-07-26T23:59:03.089744+0000","last_fullsized":"2024-07-26T23:59:03.089744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:03.088122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:03.088122+0000","last_clean_scrub_stamp":"2024-07-26T23:59:03.088122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:36:59.687635+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1864,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:03.612685+0000","last_change":"2024-07-26T23:59:03.612685+0000","last_active":"2024-07-26T23:59:03.612685+0000","last_peered":"2024-07-26T23:59:03.612685+0000","last_clean":"2024-07-26T23:59:03.612685+0000","last_became_active":"2024-07-26T23:36:45.718574+0000","last_became_peered":"2024-07-26T23:36:45.718574+0000","last_unstale":"2024-07-26T23:59:03.612685+0000","last_undegraded":"2024-07-26T23:59:03.612685+0000","last_fullsized":"2024-07-26T23:59:03.612685+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:03.610050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:03.610050+0000","last_clean_scrub_stamp":"2024-07-26T23:59:03.610050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:02:41.973576+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1864,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:57.256072+0000","last_change":"2024-07-26T23:58:57.256896+0000","last_active":"2024-07-26T23:58:57.256072+0000","last_peered":"2024-07-26T23:58:57.256072+0000","last_clean":"2024-07-26T23:58:57.256072+0000","last_became_active":"2024-07-26T23:36:33.915707+0000","last_became_peered":"2024-07-26T23:36:33.915707+0000","last_unstale":"2024-07-26T23:58:57.256072+0000","last_undegraded":"2024-07-26T23:58:57.256072+0000","last_fullsized":"2024-07-26T23:58:57.256072+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:57.254401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:57.254401+0000","last_clean_scrub_stamp":"2024-07-26T23:58:57.254401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:32:54.370467+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"970'8","reported_seq":1872,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.124977+0000","last_change":"2024-07-26T23:58:58.124977+0000","last_active":"2024-07-26T23:58:58.124977+0000","last_peered":"2024-07-26T23:58:58.124977+0000","last_clean":"2024-07-26T23:58:58.124977+0000","last_became_active":"2024-07-26T23:36:30.507651+0000","last_became_peered":"2024-07-26T23:36:30.507651+0000","last_unstale":"2024-07-26T23:58:58.124977+0000","last_undegraded":"2024-07-26T23:58:58.124977+0000","last_fullsized":"2024-07-26T23:58:58.124977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'8","last_scrub_stamp":"2024-07-26T23:58:58.122141+0000","last_deep_scrub":"970'8","last_deep_scrub_stamp":"2024-07-26T23:58:58.122141+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.122141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:57:29.495918+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1859,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.046267+0000","last_change":"2024-07-26T23:58:59.046267+0000","last_active":"2024-07-26T23:58:59.046267+0000","last_peered":"2024-07-26T23:58:59.046267+0000","last_clean":"2024-07-26T23:58:59.046267+0000","last_became_active":"2024-07-26T23:36:29.468516+0000","last_became_peered":"2024-07-26T23:36:29.468516+0000","last_unstale":"2024-07-26T23:58:59.046267+0000","last_undegraded":"2024-07-26T23:58:59.046267+0000","last_fullsized":"2024-07-26T23:58:59.046267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:59.043704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:59.043704+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.043704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:12:25.111037+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1885,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:01.212948+0000","last_change":"2024-07-26T23:59:01.212948+0000","last_active":"2024-07-26T23:59:01.212948+0000","last_peered":"2024-07-26T23:59:01.212948+0000","last_clean":"2024-07-26T23:59:01.212948+0000","last_became_active":"2024-07-26T23:36:50.085394+0000","last_became_peered":"2024-07-26T23:36:50.085394+0000","last_unstale":"2024-07-26T23:59:01.212948+0000","last_undegraded":"2024-07-26T23:59:01.212948+0000","last_fullsized":"2024-07-26T23:59:01.212948+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:01.210359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:01.210359+0000","last_clean_scrub_stamp":"2024-07-26T23:59:01.210359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:48:41.367746+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1858,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.080153+0000","last_change":"2024-07-26T23:58:58.081649+0000","last_active":"2024-07-26T23:58:58.080153+0000","last_peered":"2024-07-26T23:58:58.080153+0000","last_clean":"2024-07-26T23:58:58.080153+0000","last_became_active":"2024-07-26T23:36:32.315924+0000","last_became_peered":"2024-07-26T23:36:32.315924+0000","last_unstale":"2024-07-26T23:58:58.080153+0000","last_undegraded":"2024-07-26T23:58:58.080153+0000","last_fullsized":"2024-07-26T23:58:58.080153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:58.077277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:58.077277+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.077277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:45:51.170626+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1863,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.295828+0000","last_change":"2024-07-26T23:58:59.296709+0000","last_active":"2024-07-26T23:58:59.295828+0000","last_peered":"2024-07-26T23:58:59.295828+0000","last_clean":"2024-07-26T23:58:59.295828+0000","last_became_active":"2024-07-26T23:36:33.069066+0000","last_became_peered":"2024-07-26T23:36:33.069066+0000","last_unstale":"2024-07-26T23:58:59.295828+0000","last_undegraded":"2024-07-26T23:58:59.295828+0000","last_fullsized":"2024-07-26T23:58:59.295828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:59.294030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:59.294030+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.294030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:25:36.061049+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1881,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:01.749135+0000","last_change":"2024-07-26T23:59:01.750084+0000","last_active":"2024-07-26T23:59:01.749135+0000","last_peered":"2024-07-26T23:59:01.749135+0000","last_clean":"2024-07-26T23:59:01.749135+0000","last_became_active":"2024-07-26T23:36:32.984857+0000","last_became_peered":"2024-07-26T23:36:32.984857+0000","last_unstale":"2024-07-26T23:59:01.749135+0000","last_undegraded":"2024-07-26T23:59:01.749135+0000","last_fullsized":"2024-07-26T23:59:01.749135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:01.747296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:01.747296+0000","last_clean_scrub_stamp":"2024-07-26T23:59:01.747296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:32:33.716031+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1872,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:03.023127+0000","last_change":"2024-07-26T23:59:03.023955+0000","last_active":"2024-07-26T23:59:03.023127+0000","last_peered":"2024-07-26T23:59:03.023127+0000","last_clean":"2024-07-26T23:59:03.023127+0000","last_became_active":"2024-07-26T23:36:30.560131+0000","last_became_peered":"2024-07-26T23:36:30.560131+0000","last_unstale":"2024-07-26T23:59:03.023127+0000","last_undegraded":"2024-07-26T23:59:03.023127+0000","last_fullsized":"2024-07-26T23:59:03.023127+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:03.021504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:03.021504+0000","last_clean_scrub_stamp":"2024-07-26T23:59:03.021504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:17:20.056933+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1864,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:05.692592+0000","last_change":"2024-07-26T23:59:05.693483+0000","last_active":"2024-07-26T23:59:05.692592+0000","last_peered":"2024-07-26T23:59:05.692592+0000","last_clean":"2024-07-26T23:59:05.692592+0000","last_became_active":"2024-07-26T23:36:44.149525+0000","last_became_peered":"2024-07-26T23:36:44.149525+0000","last_unstale":"2024-07-26T23:59:05.692592+0000","last_undegraded":"2024-07-26T23:59:05.692592+0000","last_fullsized":"2024-07-26T23:59:05.692592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:05.690841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:05.690841+0000","last_clean_scrub_stamp":"2024-07-26T23:59:05.690841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:50.798032+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"970'34","reported_seq":1884,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:57.185708+0000","last_change":"2024-07-26T23:58:57.187263+0000","last_active":"2024-07-26T23:58:57.185708+0000","last_peered":"2024-07-26T23:58:57.185708+0000","last_clean":"2024-07-26T23:58:57.185708+0000","last_became_active":"2024-07-26T23:35:16.707863+0000","last_became_peered":"2024-07-26T23:35:16.707863+0000","last_unstale":"2024-07-26T23:58:57.185708+0000","last_undegraded":"2024-07-26T23:58:57.185708+0000","last_fullsized":"2024-07-26T23:58:57.185708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'34","last_scrub_stamp":"2024-07-26T23:58:57.182818+0000","last_deep_scrub":"970'34","last_deep_scrub_stamp":"2024-07-26T23:58:57.182818+0000","last_clean_scrub_stamp":"2024-07-26T23:58:57.182818+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:09:47.734483+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"970'2","reported_seq":1891,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.566927+0000","last_change":"2024-07-26T23:59:00.567946+0000","last_active":"2024-07-26T23:59:00.566927+0000","last_peered":"2024-07-26T23:59:00.566927+0000","last_clean":"2024-07-26T23:59:00.566927+0000","last_became_active":"2024-07-26T23:35:19.087174+0000","last_became_peered":"2024-07-26T23:35:19.087174+0000","last_unstale":"2024-07-26T23:59:00.566927+0000","last_undegraded":"2024-07-26T23:59:00.566927+0000","last_fullsized":"2024-07-26T23:59:00.566927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'2","last_scrub_stamp":"2024-07-26T23:59:00.565123+0000","last_deep_scrub":"970'2","last_deep_scrub_stamp":"2024-07-26T23:59:00.565123+0000","last_clean_scrub_stamp":"2024-07-26T23:59:00.565123+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:23:44.786699+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0.074002362000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1872,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:06.062678+0000","last_change":"2024-07-26T23:59:06.063571+0000","last_active":"2024-07-26T23:59:06.062678+0000","last_peered":"2024-07-26T23:59:06.062678+0000","last_clean":"2024-07-26T23:59:06.062678+0000","last_became_active":"2024-07-26T23:36:39.441853+0000","last_became_peered":"2024-07-26T23:36:39.441853+0000","last_unstale":"2024-07-26T23:59:06.062678+0000","last_undegraded":"2024-07-26T23:59:06.062678+0000","last_fullsized":"2024-07-26T23:59:06.062678+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:06.060943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:06.060943+0000","last_clean_scrub_stamp":"2024-07-26T23:59:06.060943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:42:29.885510+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1889,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.538091+0000","last_change":"2024-07-26T23:58:58.539215+0000","last_active":"2024-07-26T23:58:58.538091+0000","last_peered":"2024-07-26T23:58:58.538091+0000","last_clean":"2024-07-26T23:58:58.538091+0000","last_became_active":"2024-07-26T23:35:17.846505+0000","last_became_peered":"2024-07-26T23:35:17.846505+0000","last_unstale":"2024-07-26T23:58:58.538091+0000","last_undegraded":"2024-07-26T23:58:58.538091+0000","last_fullsized":"2024-07-26T23:58:58.538091+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:58.535053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:58.535053+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.535053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:14:14.627697+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0.11625252899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1862,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:02.584871+0000","last_change":"2024-07-26T23:59:02.584871+0000","last_active":"2024-07-26T23:59:02.584871+0000","last_peered":"2024-07-26T23:59:02.584871+0000","last_clean":"2024-07-26T23:59:02.584871+0000","last_became_active":"2024-07-26T23:36:33.958339+0000","last_became_peered":"2024-07-26T23:36:33.958339+0000","last_unstale":"2024-07-26T23:59:02.584871+0000","last_undegraded":"2024-07-26T23:59:02.584871+0000","last_fullsized":"2024-07-26T23:59:02.584871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:02.582370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:02.582370+0000","last_clean_scrub_stamp":"2024-07-26T23:59:02.582370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:42:16.281535+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1896,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:02.008811+0000","last_change":"2024-07-26T23:59:02.009752+0000","last_active":"2024-07-26T23:59:02.008811+0000","last_peered":"2024-07-26T23:59:02.008811+0000","last_clean":"2024-07-26T23:59:02.008811+0000","last_became_active":"2024-07-26T23:35:17.814359+0000","last_became_peered":"2024-07-26T23:35:17.814359+0000","last_unstale":"2024-07-26T23:59:02.008811+0000","last_undegraded":"2024-07-26T23:59:02.008811+0000","last_fullsized":"2024-07-26T23:59:02.008811+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:02.006141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:02.006141+0000","last_clean_scrub_stamp":"2024-07-26T23:59:02.006141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:06:05.275085+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0.060284956000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1873,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:07.066591+0000","last_change":"2024-07-26T23:59:07.067866+0000","last_active":"2024-07-26T23:59:07.066591+0000","last_peered":"2024-07-26T23:59:07.066591+0000","last_clean":"2024-07-26T23:59:07.066591+0000","last_became_active":"2024-07-26T23:36:32.112540+0000","last_became_peered":"2024-07-26T23:36:32.112540+0000","last_unstale":"2024-07-26T23:59:07.066591+0000","last_undegraded":"2024-07-26T23:59:07.066591+0000","last_fullsized":"2024-07-26T23:59:07.066591+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:07.064097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:07.064097+0000","last_clean_scrub_stamp":"2024-07-26T23:59:07.064097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:11.401050+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"970'3","reported_seq":1892,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.557334+0000","last_change":"2024-07-26T23:58:59.558707+0000","last_active":"2024-07-26T23:58:59.557334+0000","last_peered":"2024-07-26T23:58:59.557334+0000","last_clean":"2024-07-26T23:58:59.557334+0000","last_became_active":"2024-07-26T23:35:18.492082+0000","last_became_peered":"2024-07-26T23:35:18.492082+0000","last_unstale":"2024-07-26T23:58:59.557334+0000","last_undegraded":"2024-07-26T23:58:59.557334+0000","last_fullsized":"2024-07-26T23:58:59.557334+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'3","last_scrub_stamp":"2024-07-26T23:58:59.555471+0000","last_deep_scrub":"970'3","last_deep_scrub_stamp":"2024-07-26T23:58:59.555471+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.555471+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:40:19.263535+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0.098681633000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1884,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:02.186450+0000","last_change":"2024-07-26T23:59:02.187284+0000","last_active":"2024-07-26T23:59:02.186450+0000","last_peered":"2024-07-26T23:59:02.186450+0000","last_clean":"2024-07-26T23:59:02.186450+0000","last_became_active":"2024-07-26T23:36:29.374355+0000","last_became_peered":"2024-07-26T23:36:29.374355+0000","last_unstale":"2024-07-26T23:59:02.186450+0000","last_undegraded":"2024-07-26T23:59:02.186450+0000","last_fullsized":"2024-07-26T23:59:02.186450+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:02.184778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:02.184778+0000","last_clean_scrub_stamp":"2024-07-26T23:59:02.184778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:50:44.819761+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1894,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.863577+0000","last_change":"2024-07-26T23:59:00.864465+0000","last_active":"2024-07-26T23:59:00.863577+0000","last_peered":"2024-07-26T23:59:00.863577+0000","last_clean":"2024-07-26T23:59:00.863577+0000","last_became_active":"2024-07-26T23:35:18.507138+0000","last_became_peered":"2024-07-26T23:35:18.507138+0000","last_unstale":"2024-07-26T23:59:00.863577+0000","last_undegraded":"2024-07-26T23:59:00.863577+0000","last_fullsized":"2024-07-26T23:59:00.863577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:00.861801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:00.861801+0000","last_clean_scrub_stamp":"2024-07-26T23:59:00.861801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:59:52.761424+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0.047472209000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1873,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:05.015035+0000","last_change":"2024-07-26T23:59:05.015886+0000","last_active":"2024-07-26T23:59:05.015035+0000","last_peered":"2024-07-26T23:59:05.015035+0000","last_clean":"2024-07-26T23:59:05.015035+0000","last_became_active":"2024-07-26T23:36:32.842079+0000","last_became_peered":"2024-07-26T23:36:32.842079+0000","last_unstale":"2024-07-26T23:59:05.015035+0000","last_undegraded":"2024-07-26T23:59:05.015035+0000","last_fullsized":"2024-07-26T23:59:05.015035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:05.013253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:05.013253+0000","last_clean_scrub_stamp":"2024-07-26T23:59:05.013253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:56:54.582521+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1888,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:04.647747+0000","last_change":"2024-07-26T23:59:04.648618+0000","last_active":"2024-07-26T23:59:04.647747+0000","last_peered":"2024-07-26T23:59:04.647747+0000","last_clean":"2024-07-26T23:59:04.647747+0000","last_became_active":"2024-07-26T23:35:17.925979+0000","last_became_peered":"2024-07-26T23:35:17.925979+0000","last_unstale":"2024-07-26T23:59:04.647747+0000","last_undegraded":"2024-07-26T23:59:04.647747+0000","last_fullsized":"2024-07-26T23:59:04.647747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:04.646106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:04.646106+0000","last_clean_scrub_stamp":"2024-07-26T23:59:04.646106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T01:18:13.480841+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0.073846415999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1873,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:04.006665+0000","last_change":"2024-07-26T23:59:04.007555+0000","last_active":"2024-07-26T23:59:04.006665+0000","last_peered":"2024-07-26T23:59:04.006665+0000","last_clean":"2024-07-26T23:59:04.006665+0000","last_became_active":"2024-07-26T23:36:31.543107+0000","last_became_peered":"2024-07-26T23:36:31.543107+0000","last_unstale":"2024-07-26T23:59:04.006665+0000","last_undegraded":"2024-07-26T23:59:04.006665+0000","last_fullsized":"2024-07-26T23:59:04.006665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:04.004972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:04.004972+0000","last_clean_scrub_stamp":"2024-07-26T23:59:04.004972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:59:59.713723+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1906,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.176501+0000","last_change":"2024-07-26T23:59:00.176501+0000","last_active":"2024-07-26T23:59:00.176501+0000","last_peered":"2024-07-26T23:59:00.176501+0000","last_clean":"2024-07-26T23:59:00.176501+0000","last_became_active":"2024-07-26T23:35:18.587303+0000","last_became_peered":"2024-07-26T23:35:18.587303+0000","last_unstale":"2024-07-26T23:59:00.176501+0000","last_undegraded":"2024-07-26T23:59:00.176501+0000","last_fullsized":"2024-07-26T23:59:00.176501+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:00.174009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:00.174009+0000","last_clean_scrub_stamp":"2024-07-26T23:59:00.174009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:36:10.433424+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.077704690000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1881,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:02.728069+0000","last_change":"2024-07-26T23:59:02.728891+0000","last_active":"2024-07-26T23:59:02.728069+0000","last_peered":"2024-07-26T23:59:02.728069+0000","last_clean":"2024-07-26T23:59:02.728069+0000","last_became_active":"2024-07-26T23:36:50.258118+0000","last_became_peered":"2024-07-26T23:36:50.258118+0000","last_unstale":"2024-07-26T23:59:02.728069+0000","last_undegraded":"2024-07-26T23:59:02.728069+0000","last_fullsized":"2024-07-26T23:59:02.728069+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:02.726443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:02.726443+0000","last_clean_scrub_stamp":"2024-07-26T23:59:02.726443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:28:20.340222+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1886,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:01.563006+0000","last_change":"2024-07-26T23:59:01.563006+0000","last_active":"2024-07-26T23:59:01.563006+0000","last_peered":"2024-07-26T23:59:01.563006+0000","last_clean":"2024-07-26T23:59:01.563006+0000","last_became_active":"2024-07-26T23:35:18.556903+0000","last_became_peered":"2024-07-26T23:35:18.556903+0000","last_unstale":"2024-07-26T23:59:01.563006+0000","last_undegraded":"2024-07-26T23:59:01.563006+0000","last_fullsized":"2024-07-26T23:59:01.563006+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:01.560130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:01.560130+0000","last_clean_scrub_stamp":"2024-07-26T23:59:01.560130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:05:32.144366+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0.064553026999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1862,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:58.323480+0000","last_change":"2024-07-26T23:58:58.324317+0000","last_active":"2024-07-26T23:58:58.323480+0000","last_peered":"2024-07-26T23:58:58.323480+0000","last_clean":"2024-07-26T23:58:58.323480+0000","last_became_active":"2024-07-26T23:36:28.821361+0000","last_became_peered":"2024-07-26T23:36:28.821361+0000","last_unstale":"2024-07-26T23:58:58.323480+0000","last_undegraded":"2024-07-26T23:58:58.323480+0000","last_fullsized":"2024-07-26T23:58:58.323480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:58.321762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:58.321762+0000","last_clean_scrub_stamp":"2024-07-26T23:58:58.321762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:10:03.939499+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1857,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:01.083365+0000","last_change":"2024-07-26T23:59:01.084247+0000","last_active":"2024-07-26T23:59:01.083365+0000","last_peered":"2024-07-26T23:59:01.083365+0000","last_clean":"2024-07-26T23:59:01.083365+0000","last_became_active":"2024-07-26T23:36:29.496365+0000","last_became_peered":"2024-07-26T23:36:29.496365+0000","last_unstale":"2024-07-26T23:59:01.083365+0000","last_undegraded":"2024-07-26T23:59:01.083365+0000","last_fullsized":"2024-07-26T23:59:01.083365+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:01.081592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:01.081592+0000","last_clean_scrub_stamp":"2024-07-26T23:59:01.081592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:02:43.336544+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1872,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:09.030706+0000","last_change":"2024-07-26T23:59:09.031605+0000","last_active":"2024-07-26T23:59:09.030706+0000","last_peered":"2024-07-26T23:59:09.030706+0000","last_clean":"2024-07-26T23:59:09.030706+0000","last_became_active":"2024-07-26T23:36:33.301227+0000","last_became_peered":"2024-07-26T23:36:33.301227+0000","last_unstale":"2024-07-26T23:59:09.030706+0000","last_undegraded":"2024-07-26T23:59:09.030706+0000","last_fullsized":"2024-07-26T23:59:09.030706+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:09.029060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:09.029060+0000","last_clean_scrub_stamp":"2024-07-26T23:59:09.029060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:01:46.947462+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1872,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:08.031481+0000","last_change":"2024-07-26T23:59:08.032383+0000","last_active":"2024-07-26T23:59:08.031481+0000","last_peered":"2024-07-26T23:59:08.031481+0000","last_clean":"2024-07-26T23:59:08.031481+0000","last_became_active":"2024-07-26T23:36:30.280064+0000","last_became_peered":"2024-07-26T23:36:30.280064+0000","last_unstale":"2024-07-26T23:59:08.031481+0000","last_undegraded":"2024-07-26T23:59:08.031481+0000","last_fullsized":"2024-07-26T23:59:08.031481+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:08.029757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:08.029757+0000","last_clean_scrub_stamp":"2024-07-26T23:59:08.029757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:24:24.372999+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1854,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.080570+0000","last_change":"2024-07-26T23:59:00.081482+0000","last_active":"2024-07-26T23:59:00.080570+0000","last_peered":"2024-07-26T23:59:00.080570+0000","last_clean":"2024-07-26T23:59:00.080570+0000","last_became_active":"2024-07-26T23:36:29.640586+0000","last_became_peered":"2024-07-26T23:36:29.640586+0000","last_unstale":"2024-07-26T23:59:00.080570+0000","last_undegraded":"2024-07-26T23:59:00.080570+0000","last_fullsized":"2024-07-26T23:59:00.080570+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:00.078693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:00.078693+0000","last_clean_scrub_stamp":"2024-07-26T23:59:00.078693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:06:31.334051+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1857,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:02.052461+0000","last_change":"2024-07-26T23:59:02.053356+0000","last_active":"2024-07-26T23:59:02.052461+0000","last_peered":"2024-07-26T23:59:02.052461+0000","last_clean":"2024-07-26T23:59:02.052461+0000","last_became_active":"2024-07-26T23:36:28.852843+0000","last_became_peered":"2024-07-26T23:36:28.852843+0000","last_unstale":"2024-07-26T23:59:02.052461+0000","last_undegraded":"2024-07-26T23:59:02.052461+0000","last_fullsized":"2024-07-26T23:59:02.052461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:02.050730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:02.050730+0000","last_clean_scrub_stamp":"2024-07-26T23:59:02.050730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:21:45.611711+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1855,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:59.085677+0000","last_change":"2024-07-26T23:58:59.086559+0000","last_active":"2024-07-26T23:58:59.085677+0000","last_peered":"2024-07-26T23:58:59.085677+0000","last_clean":"2024-07-26T23:58:59.085677+0000","last_became_active":"2024-07-26T23:36:31.124146+0000","last_became_peered":"2024-07-26T23:36:31.124146+0000","last_unstale":"2024-07-26T23:58:59.085677+0000","last_undegraded":"2024-07-26T23:58:59.085677+0000","last_fullsized":"2024-07-26T23:58:59.085677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:59.082134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:59.082134+0000","last_clean_scrub_stamp":"2024-07-26T23:58:59.082134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:08:17.106823+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1862,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:58:56.252186+0000","last_change":"2024-07-26T23:58:56.253051+0000","last_active":"2024-07-26T23:58:56.252186+0000","last_peered":"2024-07-26T23:58:56.252186+0000","last_clean":"2024-07-26T23:58:56.252186+0000","last_became_active":"2024-07-26T23:36:30.173625+0000","last_became_peered":"2024-07-26T23:36:30.173625+0000","last_unstale":"2024-07-26T23:58:56.252186+0000","last_undegraded":"2024-07-26T23:58:56.252186+0000","last_fullsized":"2024-07-26T23:58:56.252186+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:58:56.250348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:58:56.250348+0000","last_clean_scrub_stamp":"2024-07-26T23:58:56.250348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T03:18:56.590554+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1870,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:01.871375+0000","last_change":"2024-07-26T23:59:01.872260+0000","last_active":"2024-07-26T23:59:01.871375+0000","last_peered":"2024-07-26T23:59:01.871375+0000","last_clean":"2024-07-26T23:59:01.871375+0000","last_became_active":"2024-07-26T23:36:31.990984+0000","last_became_peered":"2024-07-26T23:36:31.990984+0000","last_unstale":"2024-07-26T23:59:01.871375+0000","last_undegraded":"2024-07-26T23:59:01.871375+0000","last_fullsized":"2024-07-26T23:59:01.871375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:01.869599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:01.869599+0000","last_clean_scrub_stamp":"2024-07-26T23:59:01.869599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T04:22:08.872751+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1883,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:03.224137+0000","last_change":"2024-07-26T23:59:03.224965+0000","last_active":"2024-07-26T23:59:03.224137+0000","last_peered":"2024-07-26T23:59:03.224137+0000","last_clean":"2024-07-26T23:59:03.224137+0000","last_became_active":"2024-07-26T23:36:33.382005+0000","last_became_peered":"2024-07-26T23:36:33.382005+0000","last_unstale":"2024-07-26T23:59:03.224137+0000","last_undegraded":"2024-07-26T23:59:03.224137+0000","last_fullsized":"2024-07-26T23:59:03.224137+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:03.222458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:03.222458+0000","last_clean_scrub_stamp":"2024-07-26T23:59:03.222458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:35:48.092029+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1857,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:00.076468+0000","last_change":"2024-07-26T23:59:00.077364+0000","last_active":"2024-07-26T23:59:00.076468+0000","last_peered":"2024-07-26T23:59:00.076468+0000","last_clean":"2024-07-26T23:59:00.076468+0000","last_became_active":"2024-07-26T23:36:28.654711+0000","last_became_peered":"2024-07-26T23:36:28.654711+0000","last_unstale":"2024-07-26T23:59:00.076468+0000","last_undegraded":"2024-07-26T23:59:00.076468+0000","last_fullsized":"2024-07-26T23:59:00.076468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:00.074526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:00.074526+0000","last_clean_scrub_stamp":"2024-07-26T23:59:00.074526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T02:04:39.164094+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1880,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-07-26T23:59:03.731714+0000","last_change":"2024-07-26T23:59:03.732545+0000","last_active":"2024-07-26T23:59:03.731714+0000","last_peered":"2024-07-26T23:59:03.731714+0000","last_clean":"2024-07-26T23:59:03.731714+0000","last_became_active":"2024-07-26T23:36:36.803418+0000","last_became_peered":"2024-07-26T23:36:36.803418+0000","last_unstale":"2024-07-26T23:59:03.731714+0000","last_undegraded":"2024-07-26T23:59:03.731714+0000","last_fullsized":"2024-07-26T23:59:03.731714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T23:59:03.730042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T23:59:03.730042+0000","last_clean_scrub_stamp":"2024-07-26T23:59:03.730042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:34:56.399038+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":21,"seq":90194313509,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94456,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94277384,"statfs":{"total":96636764160,"available":96540041216,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313513,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90296,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281544,"statfs":{"total":96636764160,"available":96544301056,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":20,"seq":85899346214,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90300,"kb_used_data":14708,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281540,"statfs":{"total":96636764160,"available":96544296960,"internally_reserved":0,"allocated":15060992,"data_stored":9474674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346217,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90692,"kb_used_data":15100,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281148,"statfs":{"total":96636764160,"available":96543895552,"internally_reserved":0,"allocated":15462400,"data_stored":9872500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411622,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81976,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94289864,"statfs":{"total":96636764160,"available":96552820736,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509734,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90296,"kb_used_data":14704,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94281544,"statfs":{"total":96636764160,"available":96544301056,"internally_reserved":0,"allocated":15056896,"data_stored":9474655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575141,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77824,"kb_used_data":14712,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94294016,"statfs":{"total":96636764160,"available":96557072384,"internally_reserved":0,"allocated":15065088,"data_stored":9474679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575139,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94848,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94276992,"statfs":{"total":96636764160,"available":96539639808,"internally_reserved":0,"allocated":15458304,"data_stored":9872495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T23:59:23.153 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-07-26T23:59:23.255 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.240+0000 7f456e7aa640 10 monclient: get_monmap_and_config 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.240+0000 7f456e7aa640 10 monclient: build_initial_monmap 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.240+0000 7f456e7aa640 10 monclient: monmap: 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:59:23.241401+0000 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:59:23.241401+0000 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:0: v1:172.21.15.139:6789/0 mon.noname-a-legacy 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-b 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:3: v2:172.21.15.139:3300/0 mon.noname-a 2024-07-26T23:59:23.256 INFO:teuthology.orchestra.run.smithi139.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-07-26T23:59:23.257 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 1 Processor -- start 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 1 -- start start 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient: init 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f456806e720 0x7f4568139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient(hunting): picked mon.noname-c con 0x7f456806e720 addr v2:172.21.15.177:3300/0 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4568143d00 addr v1:172.21.15.177:6789/0 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4568144cd0 0x7f4568145100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient(hunting): picked mon.noname-b con 0x7f4568144cd0 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45680ac810 con 0x7f4568143d00 2024-07-26T23:59:23.258 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:23.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f456806eb00 con 0x7f456806e720 2024-07-26T23:59:23.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:23.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f4568144710 con 0x7f4568144cd0 2024-07-26T23:59:23.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:23.259 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.241+0000 7f456e7aa640 10 monclient(hunting): authenticate will time out at 2796.277790s 2024-07-26T23:59:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.271+0000 7f4567fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f456806e720 0x7f4568139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.271+0000 7f4567fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f456806e720 0x7f4568139bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:58412/0 (socket says 172.21.15.139:58412) 2024-07-26T23:59:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.271+0000 7f4567fff640 1 -- 172.21.15.139:0/4152219472 learned_addr learned my addr 172.21.15.139:0/4152219472 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.271+0000 7f4567fff640 10 monclient(hunting): get_auth_request con 0x7f456806e720 auth_method 0 2024-07-26T23:59:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.271+0000 7f4567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.271+0000 7f4567fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.271+0000 7f4567fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:23.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.272+0000 7f4567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:23.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.272+0000 7f4567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:23.288 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.272+0000 7f4567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f456cd47640 1 --2- 172.21.15.139:0/4152219472 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4568144cd0 0x7f4568145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f456cd47640 10 monclient(hunting): get_auth_request con 0x7f4568144cd0 auth_method 0 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f456cd47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f456cd47640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f456cd47640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 10 monclient(hunting): handle_auth_done global_id 20371 payload 274 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 1 -- 172.21.15.139:0/4152219472 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4568144cd0 msgr2=0x7f4568145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 1 --2- 172.21.15.139:0/4152219472 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4568144cd0 0x7f4568145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 1 -- 172.21.15.139:0/4152219472 >> v1:172.21.15.177:6789/0 conn(0x7f4568143d00 legacy=0x7f456813a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T23:59:23.289 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 10 monclient: _finish_hunting 0 2024-07-26T23:59:23.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 1 monclient: found mon.noname-c 2024-07-26T23:59:23.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:23.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-07-26T23:59:23.290 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.273+0000 7f4567fff640 1 -- 172.21.15.139:0/4152219472 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45680acaf0 con 0x7f456806e720 2024-07-26T23:59:23.292 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.277+0000 7f4567fff640 1 --2- 172.21.15.139:0/4152219472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456806e720 0x7f4568139bd0 secure :-1 s=READY pgs=4956 cs=0 l=1 rev1=1 crypto rx=0x7f4558009870 tx=0x7f455802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcb1eb3f96d28f3f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:23.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 1 -- 172.21.15.139:0/4152219472 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558034070 con 0x7f456806e720 2024-07-26T23:59:23.300 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: dump: 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: _finish_auth 0 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: _check_auth_tickets 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 1 -- 172.21.15.139:0/4152219472 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4558030840 con 0x7f456806e720 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 1 -- 172.21.15.139:0/4152219472 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558030c10 con 0x7f456806e720 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f456e7aa640 5 monclient: authenticate success, global_id 20371 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f456e7aa640 10 monclient: get_monmap_and_config success 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f456e7aa640 4 set_mon_vals no callback set 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.282+0000 7f4566ffd640 10 monclient: dump: 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 10 monclient: shutdown 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 -- 172.21.15.139:0/4152219472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456806e720 msgr2=0x7f4568139bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 --2- 172.21.15.139:0/4152219472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456806e720 0x7f4568139bd0 secure :-1 s=READY pgs=4956 cs=0 l=1 rev1=1 crypto rx=0x7f4558009870 tx=0x7f455802b980 comp rx=0 tx=0).stop 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 -- 172.21.15.139:0/4152219472 shutdown_connections 2024-07-26T23:59:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 --2- 172.21.15.139:0/4152219472 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f4568144cd0 0x7f4568145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 --2- 172.21.15.139:0/4152219472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456806e720 0x7f4568139bd0 unknown :-1 s=CLOSED pgs=4956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 -- 172.21.15.139:0/4152219472 >> 172.21.15.139:0/4152219472 conn(0x7f45680b9c10 msgr2=0x7f45680ba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 -- 172.21.15.139:0/4152219472 shutdown_connections 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 1 -- 172.21.15.139:0/4152219472 wait complete. 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 10 monclient: build_initial_monmap 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.283+0000 7f456e7aa640 10 monclient: monmap: 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 0 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 0.000000 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:created 0.000000 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 0 (unknown) 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.noname-a 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.noname-c 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 1 librados: starting msgr at 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 1 librados: starting objecter 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 1 Processor -- start 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 1 -- start start 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 1 librados: setting wanted keys 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 1 librados: calling monclient init 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 10 monclient: init 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 10 monclient: _reopen_session rank -1 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.284+0000 7f456e7aa640 10 monclient: _add_conns ranks=[2,0,1] 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f456806e720 0x7f45681406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): picked mon.noname-c con 0x7f456806e720 addr [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4568143d00 0x7f45681339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): picked mon.noname-a con 0x7f4568143d00 addr [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568144cd0 0x7f4568133ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): picked mon.noname-b con 0x7f4568144cd0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 -- 0x7f4568145640 con 0x7f4568143d00 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4568141390 con 0x7f4568144cd0 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): start opening mon connection 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 1 -- --> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] -- mon_getmap magic: 0 -- 0x7f45681457c0 con 0x7f456806e720 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): _renew_subs 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456e7aa640 10 monclient(hunting): authenticate will time out at 2796.321098s 2024-07-26T23:59:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f45677fe640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4568143d00 0x7f45681339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:23.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f4567fff640 1 --2- >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f456806e720 0x7f45681406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:23.307 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456cd47640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568144cd0 0x7f4568133ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:23.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456cd47640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568144cd0 0x7f4568133ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.139:58424/0 (socket says 172.21.15.139:58424) 2024-07-26T23:59:23.308 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456cd47640 1 -- 172.21.15.139:0/3646931436 learned_addr learned my addr 172.21.15.139:0/3646931436 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456cd47640 10 monclient(hunting): get_auth_request con 0x7f4568144cd0 auth_method 0 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456cd47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456cd47640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f456cd47640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f45677fe640 10 monclient(hunting): get_auth_request con 0x7f4568143d00 auth_method 0 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f45677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f45677fe640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.285+0000 7f45677fe640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f4567fff640 10 monclient(hunting): get_auth_request con 0x7f456806e720 auth_method 0 2024-07-26T23:59:23.309 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f4567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f4567fff640 10 monclient(hunting): _init_auth method 2 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f4567fff640 10 monclient(hunting): _init_auth creating new auth 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f456cd47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f456cd47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f456cd47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f45677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f45677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f45677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f4567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f4567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.286+0000 7f4567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 10 monclient(hunting): handle_auth_done global_id 20374 payload 824 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 1 -- 172.21.15.139:0/3646931436 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f456806e720 msgr2=0x7f45681406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:23.310 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f456806e720 0x7f45681406e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 1 -- 172.21.15.139:0/3646931436 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4568143d00 msgr2=0x7f45681339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4568143d00 0x7f45681339b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 10 monclient: _finish_hunting 0 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 1 monclient: found mon.noname-b 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.289+0000 7f456cd47640 1 -- 172.21.15.139:0/3646931436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456813f6d0 con 0x7f4568144cd0 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.291+0000 7f456cd47640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568144cd0 0x7f4568133ef0 secure :-1 s=READY pgs=4957 cs=0 l=1 rev1=1 crypto rx=0x7f4560007e60 tx=0x7f456000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.291+0000 7f4564ff9640 10 client.?.objecter ms_handle_connect 0x7f4568144cd0 2024-07-26T23:59:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.291+0000 7f4564ff9640 10 client.?.objecter resend_mon_ops 2024-07-26T23:59:23.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560014420 con 0x7f4568144cd0 2024-07-26T23:59:23.312 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: dump: 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:23.313 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: _finish_auth 0 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: _check_auth_tickets 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4560019070 con 0x7f4568144cd0 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 5 monclient: authenticate success, global_id 20374 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 10 monclient: _renew_subs 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:23.314 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f456813f9b0 con 0x7f4568144cd0 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: handle_config config(0 keys) 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560014820 con 0x7f4568144cd0 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f4564ff9640 10 monclient: dump: 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:epoch 1 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:fsid 3b7b1763-526e-44bf-8f07-302d3301e114 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:last_changed 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:created 2024-07-26T23:33:19.536099+0000 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:min_mon_release 19 (squid) 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:election_strategy: 1 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-07-26T23:59:23.315 INFO:teuthology.orchestra.run.smithi139.stderr:2: [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] mon.c 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 10 client.20374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 10 monclient: _renew_subs 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f456813ff70 con 0x7f4568144cd0 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f456e7aa640 1 librados: init done 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.295+0000 7f45667fc640 4 set_mon_vals no callback set 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.296+0000 7f456e7aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.296+0000 7f456e7aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.296+0000 7f456e7aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:23.316 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.296+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4568145100 con 0x7f4568144cd0 2024-07-26T23:59:23.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.550+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 4) ==== 99139+0+0 (secure 0 0 0) 0x7f45600149c0 con 0x7f4568144cd0 2024-07-26T23:59:23.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.550+0000 7f4564ff9640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f45540789a0 0x7f455407ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T23:59:23.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.551+0000 7f4567fff640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f45540789a0 0x7f455407ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T23:59:23.566 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.551+0000 7f4567fff640 10 monclient: get_auth_request con 0x7f45540789a0 auth_method 0 2024-07-26T23:59:23.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.551+0000 7f4564ff9640 10 client.20374.objecter ms_handle_connect 0x7f45540789a0 2024-07-26T23:59:23.567 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.551+0000 7f4567fff640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f45540789a0 0x7f455407ae60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4558036000 tx=0x7f4558008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T23:59:23.568 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.553+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(970..970 src has 735..970) ==== 6365+0+0 (secure 0 0 0) 0x7f456005ec70 con 0x7f4568144cd0 2024-07-26T23:59:23.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.553+0000 7f4564ff9640 10 client.20374.objecter ms_dispatch 0x7f4568000b90 osd_map(970..970 src has 735..970) 2024-07-26T23:59:23.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.553+0000 7f4564ff9640 3 client.20374.objecter handle_osd_map got epochs [970,970] > 0 2024-07-26T23:59:23.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.553+0000 7f4564ff9640 3 client.20374.objecter handle_osd_map decoding full epoch 970 2024-07-26T23:59:23.569 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.553+0000 7f4564ff9640 20 client.20374.objecter dump_active .. 0 homeless 2024-07-26T23:59:23.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.563+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+193565 (secure 0 0 0) 0x7f45600629e0 con 0x7f4568144cd0 2024-07-26T23:59:23.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.563+0000 7f4564ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-07-26T23:59:23.579 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.563+0000 7f4564ff9640 10 monclient: _finish_command 1 = system:0 2024-07-26T23:59:23.762 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.747+0000 7f456e7aa640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-07-26T23:59:23.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.747+0000 7f456e7aa640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-07-26T23:59:23.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.747+0000 7f456e7aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-07-26T23:59:23.763 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.747+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f45680ac250 con 0x7f4568144cd0 2024-07-26T23:59:23.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.938+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f45600624f0 con 0x7f4568144cd0 2024-07-26T23:59:23.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.938+0000 7f4564ff9640 10 monclient: handle_config config(1 keys) 2024-07-26T23:59:23.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.938+0000 7f45667fc640 4 set_mon_vals no callback set 2024-07-26T23:59:23.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.938+0000 7f45667fc640 10 set_mon_vals mon_health_to_clog = false 2024-07-26T23:59:23.954 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.938+0000 7f4564ff9640 1 -- 172.21.15.139:0/3646931436 <== mon.1 v2:172.21.15.177:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) ==== 123+0+0 (secure 0 0 0) 0x7f4560066f30 con 0x7f4568144cd0 2024-07-26T23:59:23.955 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.938+0000 7f4564ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-07-26T23:59:23.955 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.938+0000 7f4564ff9640 10 monclient: _finish_command 2 = system:0 2024-07-26T23:59:23.957 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 10 librados: watch_flush enter 2024-07-26T23:59:23.957 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 10 librados: watch_flush exit 2024-07-26T23:59:23.957 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 20 client.20374.objecter shutdown clearing up homeless session... 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 10 client.20374.objecter successfully canceled tick 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f45540789a0 msgr2=0x7f455407ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f45540789a0 0x7f455407ae60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4558036000 tx=0x7f4558008030 comp rx=0 tx=0).stop 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 10 monclient: shutdown 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568144cd0 msgr2=0x7f4568133ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568144cd0 0x7f4568133ef0 secure :-1 s=READY pgs=4957 cs=0 l=1 rev1=1 crypto rx=0x7f4560007e60 tx=0x7f456000e4b0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 shutdown_connections 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:6800/1751160634,v1:172.21.15.177:6801/1751160634] conn(0x7f45540789a0 0x7f455407ae60 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568144cd0 0x7f4568133ef0 unknown :-1 s=CLOSED pgs=4957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4568143d00 0x7f45681339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.958 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 --2- 172.21.15.139:0/3646931436 >> [v2:172.21.15.139:3301/0,v1:172.21.15.139:6790/0] conn(0x7f456806e720 0x7f45681406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T23:59:23.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 >> 172.21.15.139:0/3646931436 conn(0x7f45680b9c10 msgr2=0x7f4568146b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T23:59:23.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 shutdown_connections 2024-07-26T23:59:23.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 -- 172.21.15.139:0/3646931436 wait complete. 2024-07-26T23:59:23.959 INFO:teuthology.orchestra.run.smithi139.stderr:2024-07-26T23:59:23.941+0000 7f456e7aa640 1 librados: shutdown 2024-07-26T23:59:23.969 INFO:teuthology.misc:Shutting down mds daemons... 2024-07-26T23:59:23.969 INFO:teuthology.misc:Shutting down osd daemons... 2024-07-26T23:59:23.969 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-07-26T23:59:23.969 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T23:59:23.982 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:59:23.958+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35253) UID: 0 2024-07-26T23:59:23.982 INFO:tasks.ceph.osd.0.smithi139.stderr:2024-07-26T23:59:23.959+0000 ac84640 -1 osd.0 970 *** Got signal Terminated *** 2024-07-26T23:59:29.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-07-26T23:59:30.071 INFO:tasks.ceph.osd.0:Stopped 2024-07-26T23:59:30.072 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-07-26T23:59:30.072 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T23:59:30.085 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:59:30.061+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35258) UID: 0 2024-07-26T23:59:30.085 INFO:tasks.ceph.osd.1.smithi139.stderr:2024-07-26T23:59:30.062+0000 ac84640 -1 osd.1 973 *** Got signal Terminated *** 2024-07-26T23:59:35.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-07-26T23:59:35.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-07-26T23:59:36.173 INFO:tasks.ceph.osd.1:Stopped 2024-07-26T23:59:36.174 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-07-26T23:59:36.174 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T23:59:36.189 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:59:36.162+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35266) UID: 0 2024-07-26T23:59:36.190 INFO:tasks.ceph.osd.2.smithi139.stderr:2024-07-26T23:59:36.164+0000 ac84640 -1 osd.2 976 *** Got signal Terminated *** 2024-07-26T23:59:40.594 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:48.251 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.649 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:48.306 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.687 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:48.344 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.699 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:48.347 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.701 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:48.349 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.714 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:48.371 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.717 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:48.374 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.727 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:48.375 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:40.749 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:48.406 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.243 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:48.891 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.244 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:48.892 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.252 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:25:48.900 34902== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.272 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:48.928 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.273 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:48.930 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.284 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:25:48.941 34959== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.311 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:48.968 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.311 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:48.969 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.320 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:25:48.977 34964== Warning: unimplemented fcntl command: 1036 2024-07-26T23:59:41.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-07-27T00:00:00.696 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:08.352 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.699 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:08.356 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.712 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:08.369 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.754 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:08.401 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.755 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:08.403 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.755 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:08.412 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.757 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:08.414 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.768 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:08.416 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:00.775 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:08.432 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.717 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:13.374 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.720 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:13.377 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.734 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:13.391 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.772 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:13.429 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.773 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:13.421 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.774 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:13.431 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.775 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:13.423 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.785 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:13.442 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:05.785 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:13.433 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.764 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:28.420 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.765 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:28.422 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.778 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:28.435 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.818 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:28.475 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.820 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:28.468 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.822 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:28.469 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.822 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:28.479 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.835 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:28.483 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:20.838 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:28.495 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.774 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:33.430 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.778 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:33.434 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.788 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:33.445 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.833 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:33.480 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.834 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:33.481 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.844 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:33.492 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.861 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:33.518 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.866 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:33.522 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:25.876 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:33.532 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.814 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:48.468 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.817 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:48.473 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.835 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:48.491 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.870 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:48.526 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.871 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:48.527 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.874 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:48.521 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.875 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:48.523 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.884 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:48.540 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:40.888 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:48.536 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.826 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:53.481 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.829 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:53.486 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.840 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:26:53.496 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.883 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:53.530 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.884 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:53.531 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.894 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:26:53.542 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.896 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:53.552 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.898 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:53.554 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:00:45.913 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:26:53.569 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.866 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:08.522 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.868 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:08.524 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.882 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:08.538 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.924 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:08.571 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.926 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:08.573 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.938 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:08.594 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.939 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:08.586 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.940 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:08.596 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:00.954 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:08.610 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.888 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:13.543 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.890 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:13.546 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.900 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:13.556 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.944 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:13.600 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.945 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:13.591 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.945 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:13.601 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.945 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:13.593 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.955 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:13.611 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:05.956 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:13.603 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.697 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:14.352 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.698 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:14.353 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.710 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:14.357 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.711 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:14.366 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.711 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:14.358 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.723 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:14.370 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.761 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:14.417 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.762 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:14.419 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:06.783 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:14.438 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:12.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-07-27T00:01:13.795 INFO:tasks.ceph.osd.2:Stopped 2024-07-27T00:01:13.795 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-07-27T00:01:13.795 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:01:13.809 INFO:tasks.ceph.osd.3.smithi139.stderr:2024-07-27T00:01:13.783+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35271) UID: 0 2024-07-27T00:01:13.809 INFO:tasks.ceph.osd.3.smithi139.stderr:2024-07-27T00:01:13.784+0000 ac84640 -1 osd.3 978 *** Got signal Terminated *** 2024-07-27T00:01:18.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-07-27T00:01:18.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-07-27T00:01:19.896 INFO:tasks.ceph.osd.3:Stopped 2024-07-27T00:01:19.897 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-07-27T00:01:19.897 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:01:19.911 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-27T00:01:19.885+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35436) UID: 0 2024-07-27T00:01:19.911 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-07-27T00:01:19.886+0000 ac84640 -1 osd.4 980 *** Got signal Terminated *** 2024-07-27T00:01:24.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-07-27T00:01:25.947 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:33.602 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:25.949 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:33.605 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:25.971 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:33.626 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.001 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:33.656 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.004 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:33.651 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.006 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:33.662 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.021 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:33.667 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.027 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:33.683 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.045 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:33.692 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.731 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:34.387 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.732 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:34.388 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.741 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:34.396 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.742 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:34.398 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.744 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:34.400 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.755 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:34.411 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.759 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:34.406 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.761 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:34.408 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:26.772 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:34.418 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.004 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:53.659 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.006 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:53.661 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.025 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:53.680 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.060 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:53.715 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.062 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:53.718 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.077 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:53.723 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.079 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:53.726 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.080 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:53.735 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.098 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:53.744 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.712 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:54.367 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.713 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:54.368 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.717 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:54.373 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.719 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:54.374 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.721 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:27:54.377 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.727 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:27:54.382 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.753 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:54.399 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.754 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:54.400 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:01:46.762 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:27:54.408 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.059 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:13.714 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.064 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:13.719 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.089 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:13.744 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.125 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:13.780 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.126 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:13.772 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.128 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:13.775 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.134 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:13.786 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.144 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:13.791 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.147 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:13.802 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.803 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:14.458 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.804 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:14.459 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.807 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:14.453 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.808 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:14.454 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.808 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:14.463 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.809 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:14.464 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.813 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:14.468 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.815 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:14.462 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:06.819 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:14.473 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.137 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:33.791 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.140 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:33.795 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.165 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:33.820 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.201 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:33.847 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.204 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:33.859 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.210 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:33.863 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.217 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:33.863 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.224 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:33.879 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:26.233 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:33.879 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.153 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:38.808 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.157 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:38.812 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.168 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:38.823 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.211 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:38.866 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.212 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:38.858 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.213 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:38.859 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.215 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:38.870 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.223 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:38.869 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:31.225 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:38.880 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.195 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:53.849 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.197 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:53.852 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.211 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:53.865 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.254 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:53.908 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.258 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:53.912 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.266 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:53.911 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.272 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:53.918 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.278 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:53.933 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:46.285 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:53.931 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.212 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:58.866 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.219 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:58.873 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.239 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:28:58.893 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.272 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:58.918 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.275 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:58.929 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.279 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:58.925 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.281 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:58.936 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.292 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:28:58.947 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:02:51.294 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:28:58.940 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.270 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:13.923 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.276 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:13.930 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.297 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:13.951 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.325 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:13.970 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.335 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:13.981 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.339 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:13.993 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.341 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:13.995 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.354 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:13.999 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:06.355 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:14.009 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.271 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:18.925 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.280 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:18.935 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.295 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:18.949 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.329 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:18.974 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.331 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:18.985 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.333 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:18.988 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.334 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:18.980 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.347 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:19.001 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:11.349 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:18.994 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.319 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:33.972 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.321 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:33.975 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.335 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:33.989 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.376 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:34.021 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.377 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:34.022 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.380 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:34.034 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.381 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:34.035 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.393 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:34.038 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:26.396 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:34.050 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.285 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:38.939 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.343 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:38.997 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.354 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:39.008 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.391 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:39.037 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.393 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:39.038 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.403 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:39.048 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.404 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:39.055 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.404 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:39.057 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:31.413 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:39.068 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.338 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:53.992 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.396 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:54.050 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.410 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:54.064 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.448 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:54.102 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.451 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:54.096 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.453 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:54.107 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.455 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:54.100 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.468 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:54.113 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:46.468 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:54.122 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.339 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:58.993 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.397 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:59.051 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.408 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:29:59.062 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.451 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:59.105 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.454 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:59.108 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.454 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:59.099 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.455 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:59.100 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.465 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:29:59.119 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:03:51.465 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:29:59.110 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.431 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:14.084 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.434 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:14.088 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.448 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:14.102 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.487 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:14.141 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.488 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:14.132 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.489 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:14.134 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.493 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:14.147 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.502 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:14.147 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:06.508 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:14.162 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.396 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:19.049 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.449 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:19.094 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.450 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:19.103 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.452 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:19.097 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.463 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:19.107 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.465 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:19.119 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.506 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:19.159 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.510 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:19.164 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:11.522 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:19.175 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.435 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:34.087 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.492 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:34.145 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.506 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:34.159 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.543 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:34.188 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.545 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:34.190 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.560 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:34.205 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.572 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:34.225 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.579 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:34.232 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:26.594 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:34.248 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.508 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:39.161 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.510 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:39.164 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.525 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:39.178 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.558 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:39.202 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.559 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:39.203 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.566 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:39.220 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.569 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:39.214 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.570 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:39.224 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:31.585 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:39.238 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.666 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:54.318 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.667 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:54.320 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.693 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:54.346 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.708 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.352 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.712 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.357 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.731 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.375 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.732 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:54.385 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.733 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:54.386 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:46.751 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:54.404 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.229 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:54.883 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.230 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:54.884 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.245 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:54.898 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.247 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.892 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.248 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.893 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.261 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:54.905 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.275 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:54.928 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.276 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:54.929 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:47.290 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:54.943 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.622 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:59.274 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.624 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:59.277 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.634 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:30:59.287 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.673 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:59.317 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.674 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:59.319 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.676 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:59.329 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.677 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:59.330 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.684 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:30:59.329 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:04:51.687 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:30:59.340 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.667 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:14.320 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.671 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:14.324 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.686 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:14.339 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.728 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.372 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.728 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:14.381 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.729 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.373 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.729 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:14.382 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.742 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:14.386 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:06.743 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:14.396 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:11.750 INFO:tasks.ceph.osd.4:Stopped 2024-07-27T00:05:11.750 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-07-27T00:05:11.750 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:05:11.764 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-07-27T00:05:11.735+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35438) UID: 0 2024-07-27T00:05:11.764 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-07-27T00:05:11.736+0000 ac84640 -1 osd.5 982 *** Got signal Terminated *** 2024-07-27T00:05:16.644 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:24.296 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.702 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:24.355 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.719 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:24.372 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.762 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:24.406 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.763 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:24.407 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.780 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:24.433 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.781 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:24.424 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.781 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:24.435 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:16.797 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:24.450 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:17.852 INFO:tasks.ceph.osd.5:Stopped 2024-07-27T00:05:17.853 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-07-27T00:05:17.853 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:05:17.865 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-07-27T00:05:17.838+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35442) UID: 0 2024-07-27T00:05:17.866 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-07-27T00:05:17.839+0000 ac84640 -1 osd.6 984 *** Got signal Terminated *** 2024-07-27T00:05:31.735 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:39.388 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.739 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:39.392 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.742 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:39.386 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.744 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:39.388 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.755 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:39.408 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.758 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:39.402 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.794 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:39.447 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.795 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:39.448 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:31.816 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:39.469 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.702 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:44.355 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.754 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:44.407 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.765 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:44.418 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.804 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:44.448 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.805 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:44.449 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.808 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:44.461 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.809 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:44.462 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.815 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:44.459 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:36.820 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:44.473 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.798 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:59.451 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.800 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:59.453 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.814 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:31:59.466 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.852 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:59.495 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.853 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:59.497 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.857 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:59.510 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.859 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:59.512 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.866 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:31:59.510 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:51.873 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:31:59.526 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:54.463 INFO:tasks.ceph.osd.6:Stopped 2024-07-27T00:05:54.463 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-07-27T00:05:54.463 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:05:54.477 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-07-27T00:05:54.448+0000 ac84640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35444) UID: 0 2024-07-27T00:05:54.477 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-07-27T00:05:54.449+0000 ac84640 -1 osd.7 984 *** Got signal Terminated *** 2024-07-27T00:05:54.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-07-27T00:05:56.813 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:32:04.466 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.817 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:32:04.469 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.828 INFO:tasks.ceph.mon.a.smithi139.stderr:==00:00:32:04.481 34959== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.871 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:32:04.524 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.873 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:32:04.526 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.874 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:04.517 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.875 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:04.519 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.884 INFO:tasks.ceph.mon.c.smithi139.stderr:==00:00:32:04.536 34964== Warning: unimplemented fcntl command: 1036 2024-07-27T00:05:56.885 INFO:tasks.ceph.mon.b.smithi177.stderr:==00:00:32:04.529 34902== Warning: unimplemented fcntl command: 1036 2024-07-27T00:06:00.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-07-27T00:06:00.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-07-27T00:06:00.565 INFO:tasks.ceph.osd.7:Stopped 2024-07-27T00:06:00.566 INFO:teuthology.misc:Shutting down mgr daemons... 2024-07-27T00:06:00.566 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-07-27T00:06:00.566 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:06:00.592 INFO:tasks.ceph.mgr.y:Stopped 2024-07-27T00:06:00.592 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-07-27T00:06:00.592 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:06:00.593 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-07-27T00:06:00.573+0000 7f702eb2c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34900) UID: 0 2024-07-27T00:06:00.593 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-07-27T00:06:00.573+0000 7f702eb2c640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-07-27T00:06:00.620 INFO:tasks.ceph.mgr.x:Stopped 2024-07-27T00:06:00.620 INFO:teuthology.misc:Shutting down mon daemons... 2024-07-27T00:06:00.620 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-07-27T00:06:00.620 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:06:00.648 INFO:tasks.ceph.mon.a.smithi139.stderr:2024-07-27T00:06:00.617+0000 9fea640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34932) UID: 0 2024-07-27T00:06:00.648 INFO:tasks.ceph.mon.a.smithi139.stderr:2024-07-27T00:06:00.619+0000 9fea640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-27T00:06:05.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-07-27T00:06:05.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-07-27T00:06:06.726 INFO:tasks.ceph.mon.a:Stopped 2024-07-27T00:06:06.727 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-07-27T00:06:06.727 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:06:06.742 INFO:tasks.ceph.mon.c.smithi139.stderr:2024-07-27T00:06:06.711+0000 9fea640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34934) UID: 0 2024-07-27T00:06:06.743 INFO:tasks.ceph.mon.c.smithi139.stderr:2024-07-27T00:06:06.713+0000 9fea640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-07-27T00:06:10.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-07-27T00:06:10.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-07-27T00:06:12.829 INFO:tasks.ceph.mon.c:Stopped 2024-07-27T00:06:12.830 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-07-27T00:06:12.831 INFO:teuthology.orchestra.run:waiting for 300 2024-07-27T00:06:12.849 INFO:tasks.ceph.mon.b.smithi177.stderr:2024-07-27T00:06:12.815+0000 9fea640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34897) UID: 0 2024-07-27T00:06:12.849 INFO:tasks.ceph.mon.b.smithi177.stderr:2024-07-27T00:06:12.816+0000 9fea640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-07-27T00:06:15.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-07-27T00:11:11.908 INFO:tasks.ceph:Checking cluster log for badness... 2024-07-27T00:11:11.909 DEBUG:teuthology.orchestra.run.smithi139:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-07-27T00:11:11.948 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-27T00:11:11.949 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-07-27T00:11:12.060 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-27T00:11:12.061 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-07-27T00:11:12.112 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-27T00:11:12.112 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-07-27T00:11:12.171 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi139.front.sepia.ceph.com 2024-07-27T00:11:12.171 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-07-27T00:11:12.236 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-27T00:11:12.237 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-07-27T00:11:12.387 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-27T00:11:12.387 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-07-27T00:11:12.444 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-27T00:11:12.445 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-07-27T00:11:12.502 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi177.front.sepia.ceph.com 2024-07-27T00:11:12.503 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-07-27T00:11:12.561 INFO:tasks.ceph:Archiving mon data... 2024-07-27T00:11:12.562 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/data/mon.a.tgz 2024-07-27T00:11:12.563 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-07-27T00:11:12.580 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.0v7A95uIh5 2024-07-27T00:11:12.581 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0v7A95uIh5 2024-07-27T00:11:12.746 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.0v7A95uIh5 2024-07-27T00:11:12.829 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.0v7A95uIh5 is 488KB 2024-07-27T00:11:12.931 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.0v7A95uIh5 2024-07-27T00:11:12.947 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/data/mon.c.tgz 2024-07-27T00:11:12.947 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-07-27T00:11:13.001 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.4Tvz1iHYwG 2024-07-27T00:11:13.002 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4Tvz1iHYwG 2024-07-27T00:11:13.153 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.4Tvz1iHYwG 2024-07-27T00:11:13.235 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.4Tvz1iHYwG is 487KB 2024-07-27T00:11:13.334 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.4Tvz1iHYwG 2024-07-27T00:11:13.349 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/data/mon.b.tgz 2024-07-27T00:11:13.350 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-07-27T00:11:13.365 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.I9Ap4MvZ2r 2024-07-27T00:11:13.366 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.I9Ap4MvZ2r 2024-07-27T00:11:13.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.I9Ap4MvZ2r 2024-07-27T00:11:13.624 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.I9Ap4MvZ2r is 487KB 2024-07-27T00:11:13.720 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.I9Ap4MvZ2r 2024-07-27T00:11:13.734 INFO:tasks.ceph:Cleaning ceph cluster... 2024-07-27T00:11:13.735 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-07-27T00:11:13.737 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-07-27T00:11:14.142 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-07-27T00:11:14.216 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 53, column 1 (, line 53) 2024-07-27T00:11:14.402 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-07-27T00:11:14.402 INFO:tasks.ceph:Archiving crash dumps... 2024-07-27T00:11:14.419 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/remote/smithi139/crash 2024-07-27T00:11:14.420 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-07-27T00:11:14.449 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/remote/smithi177/crash 2024-07-27T00:11:14.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-07-27T00:11:14.474 INFO:tasks.ceph:Compressing logs... 2024-07-27T00:11:14.474 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-07-27T00:11:14.494 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-07-27T00:11:14.517 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41054.log 2024-07-27T00:11:14.517 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2024-07-27T00:11:14.517 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41311.log 2024-07-27T00:11:14.517 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38930.log 2024-07-27T00:11:14.541 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49976.log 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40045.log 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.38930.log: /var/log/ceph/ceph-client.admin.49976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40934.log 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40045.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38930.log.gz 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42985.log 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40934.log: gzip -5 --verbose 87.5% -- /var/log/ceph/ceph-client.admin.41409.log 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40045.log.gz 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41409.log: gzip 90.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.42988.log 2024-07-27T00:11:14.542 INFO:teuthology.orchestra.run.smithi139.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.38874.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41311.log.gz 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.49976.log.gz 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39560.log 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41688.log 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39540.log 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108306.log 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39560.log.gz 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42985.log.gz 2024-07-27T00:11:14.543 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39540.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.40934.log.gz 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39311.log 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39540.log.gz 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41054.log.gz 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38770.log 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108306.log.gz 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr:gzip/var/log/ceph/ceph-client.admin.39884.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.108642.log 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.38770.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39311.log.gz 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41688.log.gz 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42988.log.gz 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2024-07-27T00:11:14.544 INFO:teuthology.orchestra.run.smithi139.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38770.log.gz 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:gzip/var/log/ceph/ceph-client.admin.108642.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42783.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42719.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108642.log.gz 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42783.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41409.log.gz 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62895.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35276.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.62895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.35276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107928.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39536.log 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35276.log.gz 2024-07-27T00:11:14.545 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.107928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39536.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107928.log.gz 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108348.log 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42719.log.gz 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.38832.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42783.log.gz 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39536.log.gz 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.62895.log.gz 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108348.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29164.log 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108724.log 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108348.log.gz 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph.tmp-client.admin.29164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41345.log 2024-07-27T00:11:14.546 INFO:teuthology.orchestra.run.smithi139.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29164.log.gz 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42989.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108724.log.gz 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.38953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42987.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42987.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41345.log.gz 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41614.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.42989.log.gz 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41228.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39736.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2024-07-27T00:11:14.547 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39736.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.42987.log.gz 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39736.log.gz 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42718.log 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39309.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108096.log 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41352.log 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40066.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108096.log.gz 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39303.log 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.42718.log.gz 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41352.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-07-27T00:11:14.548 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5/var/log/ceph/ceph-client.admin.39303.log: --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40535.log 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39303.log.gz 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.41228.log.gz 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39994.log 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107844.log 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39994.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39994.log.gz 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41352.log.gz 2024-07-27T00:11:14.549 INFO:teuthology.orchestra.run.smithi139.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2024-07-27T00:11:14.556 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-07-27T00:11:14.556 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.107844.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-07-27T00:11:14.556 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107844.log.gz 2024-07-27T00:11:14.557 INFO:teuthology.orchestra.run.smithi139.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39046.log 2024-07-27T00:11:14.557 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40514.log 2024-07-27T00:11:14.557 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39046.log: /var/log/ceph/ceph-mgr.y.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39046.log.gz 2024-07-27T00:11:14.557 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108600.log 2024-07-27T00:11:14.557 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40514.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-07-27T00:11:14.562 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-07-27T00:11:14.562 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35307.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35347.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35187.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35347.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35307.log.gz 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35187.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35347.log.gz 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35187.log.gz 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35227.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35227.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-07-27T00:11:14.563 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35227.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mgr.x.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr: 56.4% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/osd.4.log: 64.2% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr: 64.2% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.7.log: gzip 64.2% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35067.log 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/valgrind/osd.5.log: 64.2% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-07-27T00:11:14.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35267.log 2024-07-27T00:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.6.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.35067.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.35147.log 2024-07-27T00:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-27T00:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35107.log 2024-07-27T00:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35267.log: /var/log/ceph/ceph-client.admin.35147.log: 87.9% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35267.log.gz 2024-07-27T00:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35147.log.gz 2024-07-27T00:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-07-27T00:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.35107.log: /var/log/ceph/ceph-mon.b.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35107.log.gz 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107802.log 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108600.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108600.log.gz 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.107802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-osd.0.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107802.log.gz 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40762.log: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-07-27T00:11:14.569 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42717.log 2024-07-27T00:11:14.570 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41590.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-07-27T00:11:14.570 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39130.log 2024-07-27T00:11:14.573 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42717.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42717.log.gz 2024-07-27T00:11:14.573 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log 2024-07-27T00:11:14.574 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39130.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39130.log.gz 2024-07-27T00:11:14.575 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40552.log 2024-07-27T00:11:14.582 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40937.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40514.log.gz 2024-07-27T00:11:14.582 INFO:teuthology.orchestra.run.smithi139.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gz 2024-07-27T00:11:14.583 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41531.log 2024-07-27T00:11:14.583 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108138.log 2024-07-27T00:11:14.583 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz/var/log/ceph/ceph-client.admin.41531.log: 2024-07-27T00:11:14.583 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-07-27T00:11:14.583 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108138.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108138.log.gz 2024-07-27T00:11:14.583 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108054.log 2024-07-27T00:11:14.587 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-mon.c.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40552.log.gz 2024-07-27T00:11:14.587 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-07-27T00:11:14.587 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108054.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108054.log.gz 2024-07-27T00:11:14.588 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-07-27T00:11:14.588 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/valgrind/osd.0.log: 64.2% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-07-27T00:11:14.589 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-07-27T00:11:14.589 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/valgrind/mon.a.log: 74.0% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-07-27T00:11:14.589 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-07-27T00:11:14.591 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/valgrind/osd.3.log: 64.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-07-27T00:11:14.591 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-07-27T00:11:14.591 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/valgrind/osd.1.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.41531.log.gz 2024-07-27T00:11:14.591 INFO:teuthology.orchestra.run.smithi139.stderr: 64.2% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-07-27T00:11:14.591 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-07-27T00:11:14.593 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62893.log 2024-07-27T00:11:14.593 INFO:teuthology.orchestra.run.smithi139.stderr: 73.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz/var/log/ceph/valgrind/osd.2.log: 2024-07-27T00:11:14.593 INFO:teuthology.orchestra.run.smithi139.stderr: 64.2% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-07-27T00:11:14.593 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42781.log 2024-07-27T00:11:14.593 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.62893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108180.log 2024-07-27T00:11:14.597 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42781.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.62893.log.gz 2024-07-27T00:11:14.597 INFO:teuthology.orchestra.run.smithi139.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42781.log.gz 2024-07-27T00:11:14.598 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2024-07-27T00:11:14.598 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108180.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-07-27T00:11:14.598 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41624.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108180.log.gz 2024-07-27T00:11:14.598 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34974.log 2024-07-27T00:11:14.604 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-mon.a.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40535.log.gz 2024-07-27T00:11:14.610 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41306.log 2024-07-27T00:11:14.611 INFO:teuthology.orchestra.run.smithi139.stderr: 90.7%/var/log/ceph/ceph-client.admin.34974.log: -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2024-07-27T00:11:14.611 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108012.log 2024-07-27T00:11:14.613 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41306.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41306.log.gz 2024-07-27T00:11:14.613 INFO:teuthology.orchestra.run.smithi139.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.34974.log.gz 2024-07-27T00:11:14.613 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108222.log 2024-07-27T00:11:14.613 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-07-27T00:11:14.614 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108222.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108012.log.gz 2024-07-27T00:11:14.615 INFO:teuthology.orchestra.run.smithi139.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108222.log.gz 2024-07-27T00:11:14.615 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108558.log 2024-07-27T00:11:14.626 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35312.log 2024-07-27T00:11:14.627 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108558.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108558.log.gz 2024-07-27T00:11:14.627 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-07-27T00:11:14.628 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.35312.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35312.log.gz 2024-07-27T00:11:14.635 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49974.log 2024-07-27T00:11:14.646 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41091.log: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2024-07-27T00:11:14.649 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.49974.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-07-27T00:11:14.649 INFO:teuthology.orchestra.run.smithi139.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49974.log.gz 2024-07-27T00:11:14.654 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-07-27T00:11:14.656 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40222.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2024-07-27T00:11:14.656 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40815.log 2024-07-27T00:11:14.657 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41355.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-07-27T00:11:14.657 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49972.log 2024-07-27T00:11:14.670 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40815.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-07-27T00:11:14.670 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108390.log 2024-07-27T00:11:14.673 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.49972.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.49972.log.gz 2024-07-27T00:11:14.673 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2024-07-27T00:11:14.676 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108390.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108390.log.gz 2024-07-27T00:11:14.676 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-07-27T00:11:14.676 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40601.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2024-07-27T00:11:14.676 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108684.log 2024-07-27T00:11:14.685 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41455.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-07-27T00:11:14.685 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41123.log 2024-07-27T00:11:14.685 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108684.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108684.log.gz 2024-07-27T00:11:14.686 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41562.log 2024-07-27T00:11:14.686 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41123.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41123.log.gz 2024-07-27T00:11:14.687 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108474.log 2024-07-27T00:11:14.692 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41562.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41562.log.gz 2024-07-27T00:11:14.692 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40630.log 2024-07-27T00:11:14.692 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108474.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108474.log.gz 2024-07-27T00:11:14.692 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41671.log 2024-07-27T00:11:14.706 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2024-07-27T00:11:14.712 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-07-27T00:11:14.720 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41106.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.41671.log.gz 2024-07-27T00:11:14.730 INFO:teuthology.orchestra.run.smithi139.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz 2024-07-27T00:11:14.734 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42986.log 2024-07-27T00:11:14.738 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40635.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-07-27T00:11:14.747 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39308.log 2024-07-27T00:11:14.747 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42986.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.42986.log.gz 2024-07-27T00:11:14.755 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49973.log 2024-07-27T00:11:14.755 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39308.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39308.log.gz 2024-07-27T00:11:14.772 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2024-07-27T00:11:14.775 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.49973.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49973.log.gz 2024-07-27T00:11:14.775 INFO:teuthology.orchestra.run.smithi139.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40630.log.gz 2024-07-27T00:11:14.776 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40696.log 2024-07-27T00:11:14.785 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107970.log 2024-07-27T00:11:14.796 INFO:teuthology.orchestra.run.smithi177.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-07-27T00:11:14.812 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40696.log: 93.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40138.log 2024-07-27T00:11:14.813 INFO:teuthology.orchestra.run.smithi139.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-07-27T00:11:14.822 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.107970.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39004.log 2024-07-27T00:11:14.822 INFO:teuthology.orchestra.run.smithi139.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107970.log.gz 2024-07-27T00:11:14.823 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40138.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gz 2024-07-27T00:11:14.836 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42782.log 2024-07-27T00:11:14.837 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39004.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39004.log.gz 2024-07-27T00:11:14.852 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-07-27T00:11:14.854 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.42782.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42782.log.gz 2024-07-27T00:11:14.860 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39299.log 2024-07-27T00:11:14.880 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40984.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-07-27T00:11:14.881 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36112.log 2024-07-27T00:11:14.881 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39299.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39299.log.gz 2024-07-27T00:11:14.885 INFO:teuthology.orchestra.run.smithi139.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.40696.log.gz 2024-07-27T00:11:14.885 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49975.log 2024-07-27T00:11:14.886 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.36112.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36112.log.gz 2024-07-27T00:11:14.887 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39686.log 2024-07-27T00:11:14.890 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.49975.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.49975.log.gz 2024-07-27T00:11:14.890 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39904.log 2024-07-27T00:11:14.892 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39686.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39686.log.gz 2024-07-27T00:11:14.892 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37076.log 2024-07-27T00:11:14.893 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39904.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39904.log.gz 2024-07-27T00:11:14.893 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39305.log 2024-07-27T00:11:14.895 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.37076.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37076.log.gz 2024-07-27T00:11:14.895 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41674.log 2024-07-27T00:11:14.897 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39305.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39305.log.gz 2024-07-27T00:11:14.897 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108432.log 2024-07-27T00:11:14.915 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40654.log 2024-07-27T00:11:14.916 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108432.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108432.log.gz 2024-07-27T00:11:14.934 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-07-27T00:11:14.951 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40654.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40654.log.gz 2024-07-27T00:11:14.955 INFO:teuthology.orchestra.run.smithi139.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41674.log.gz 2024-07-27T00:11:14.955 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39844.log 2024-07-27T00:11:14.960 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39306.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-07-27T00:11:14.960 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35411.log 2024-07-27T00:11:14.960 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39844.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39844.log.gz 2024-07-27T00:11:14.960 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39666.log 2024-07-27T00:11:14.960 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.35411.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35411.log.gz 2024-07-27T00:11:14.961 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39537.log 2024-07-27T00:11:14.964 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39666.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39666.log.gz 2024-07-27T00:11:14.964 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2024-07-27T00:11:14.964 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39537.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39537.log.gz 2024-07-27T00:11:14.964 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40849.log 2024-07-27T00:11:14.966 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41526.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2024-07-27T00:11:14.966 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108516.log 2024-07-27T00:11:14.979 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40849.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62894.log 2024-07-27T00:11:14.981 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.108516.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108516.log.gz 2024-07-27T00:11:14.981 INFO:teuthology.orchestra.run.smithi139.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2024-07-27T00:11:14.981 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39088.log 2024-07-27T00:11:14.984 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.62894.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.62894.log.gz 2024-07-27T00:11:14.984 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-07-27T00:11:14.985 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39088.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39088.log.gz 2024-07-27T00:11:14.986 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40765.log 2024-07-27T00:11:14.987 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39735.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2024-07-27T00:11:14.987 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39312.log 2024-07-27T00:11:14.996 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40765.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40765.log.gz 2024-07-27T00:11:14.996 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-07-27T00:11:14.997 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39312.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39312.log.gz 2024-07-27T00:11:14.997 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41482.log 2024-07-27T00:11:14.999 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.39964.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-07-27T00:11:14.999 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107886.log 2024-07-27T00:11:15.008 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.41482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38340.log 2024-07-27T00:11:15.014 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.107886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-07-27T00:11:15.015 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.38340.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107886.log.gz 2024-07-27T00:11:15.029 INFO:teuthology.orchestra.run.smithi139.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38340.log.gz 2024-07-27T00:11:15.029 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2024-07-27T00:11:15.029 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40180.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-07-27T00:11:15.036 INFO:teuthology.orchestra.run.smithi139.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.41482.log.gz 2024-07-27T00:11:15.037 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108264.log 2024-07-27T00:11:15.057 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/ceph-client.admin.40668.log: /var/log/ceph/ceph-client.admin.108264.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108264.log.gz 2024-07-27T00:11:15.190 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40815.log.gz 2024-07-27T00:11:16.383 INFO:teuthology.orchestra.run.smithi139.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2024-07-27T00:11:16.427 INFO:teuthology.orchestra.run.smithi139.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-07-27T00:11:16.699 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2024-07-27T00:11:16.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-07-27T00:11:16.730 INFO:teuthology.orchestra.run.smithi177.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-07-27T00:11:17.633 INFO:teuthology.orchestra.run.smithi139.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-07-27T00:11:19.359 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-07-27T00:11:19.418 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-07-27T00:11:19.425 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-07-27T00:11:19.537 INFO:teuthology.orchestra.run.smithi177.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-07-27T00:11:20.083 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-07-27T00:11:20.125 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-07-27T00:11:20.232 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-07-27T00:11:20.235 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-07-27T00:11:20.235 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m5.727s 2024-07-27T00:11:20.235 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m28.121s 2024-07-27T00:11:20.235 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m1.026s 2024-07-27T00:11:20.411 INFO:teuthology.orchestra.run.smithi177.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-07-27T00:11:20.414 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-07-27T00:11:20.414 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m5.882s 2024-07-27T00:11:20.414 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m22.284s 2024-07-27T00:11:20.414 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.759s 2024-07-27T00:11:20.414 INFO:tasks.ceph:Archiving logs... 2024-07-27T00:11:20.415 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/remote/smithi139/log 2024-07-27T00:11:20.415 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2024-07-27T00:11:21.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~5s 2024-07-27T00:11:24.311 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/remote/smithi177/log 2024-07-27T00:11:24.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-07-27T00:11:26.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~10s 2024-07-27T00:11:27.538 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa/tasks/ceph.py", line 1973, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa/tasks/ceph.py", line 1506, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/misc.py", line 1158, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-07-27T00:11:27.540 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-07-27T00:11:27.593 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa/tasks/ceph.py", line 1973, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_67f28c2a04ad1fe90ea612a39329cc6af2a92d8b/qa/tasks/ceph.py", line 1506, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/misc.py", line 1158, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-07-27T00:11:27.594 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-07-27T00:11:27.594 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-07-27T00:11:27.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-07-27T00:11:27.675 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-07-27T00:11:27.773 INFO:teuthology.task.clock:Checking final clock skew... 2024-07-27T00:11:27.773 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T00:11:27.776 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T00:11:27.791 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-07-27T00:11:27.793 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-07-27T00:11:27.797 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T00:11:27.797 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-07-27T00:11:27.797 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +430us[ +456us] +/- 46ms 2024-07-27T00:11:27.797 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -627us[ -627us] +/- 62ms 2024-07-27T00:11:27.797 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +130us[ +130us] +/- 74ms 2024-07-27T00:11:27.797 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T00:11:27.799 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T00:11:27.799 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-07-27T00:11:27.800 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv01.front.sepia.ceph.com 3 6 377 53 +400us[ +393us] +/- 46ms 2024-07-27T00:11:27.800 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -644us[ -644us] +/- 62ms 2024-07-27T00:11:27.800 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +104us[ +104us] +/- 74ms 2024-07-27T00:11:27.800 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T00:11:27.800 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-07-27T00:11:27.807 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-07-27T00:11:27.808 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-07-27T00:11:27.845 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-07-27T00:11:27.915 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-07-27T00:11:27.984 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T00:11:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T00:11:28.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T00:11:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T00:11:28.055 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-07-27T00:11:28.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-07-27T00:11:28.095 INFO:teuthology.task.internal:Duration was 3188.966839 seconds 2024-07-27T00:11:28.096 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-07-27T00:11:28.128 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-07-27T00:11:28.129 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-07-27T00:11:28.132 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-07-27T00:11:28.174 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T00:11:28.175 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T00:11:28.343 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-07-27T00:11:28.344 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-07-27T00:11:28.344 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-07-27T00:11:28.366 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-07-27T00:11:28.366 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-07-27T00:11:28.405 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-07-27T00:11:28.405 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-07-27T00:11:28.409 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-07-27T00:11:28.473 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-07-27T00:11:28.474 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-07-27T00:11:28.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-07-27T00:11:28.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-07-27T00:11:28.625 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-07-27T00:11:28.626 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-07-27T00:11:28.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-07-27T00:11:28.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-07-27T00:11:28.688 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-07-27T00:11:28.693 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-07-27T00:11:28.716 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-07-27T00:11:28.739 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-07-27T00:11:28.758 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-07-27T00:11:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T00:11:28.786 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-07-27T00:11:28.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T00:11:28.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-07-27T00:11:28.822 INFO:teuthology.task.internal:Transferring archived files... 2024-07-27T00:11:28.823 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/remote/smithi139 2024-07-27T00:11:28.823 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-07-27T00:11:28.873 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-07-26_21:23:43-rados-wip-yuri-testing-2024-07-26-0628-distro-default-smithi/7820516/remote/smithi177 2024-07-27T00:11:28.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-07-27T00:11:28.906 INFO:teuthology.task.internal:Removing archive directory... 2024-07-27T00:11:28.906 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-07-27T00:11:28.908 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-07-27T00:11:28.957 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-07-27T00:11:28.971 INFO:teuthology.task.internal:Not uploading archives. 2024-07-27T00:11:28.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-07-27T00:11:29.031 INFO:teuthology.task.internal:Tidying up after the test... 2024-07-27T00:11:29.032 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-07-27T00:11:29.035 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-07-27T00:11:29.047 INFO:teuthology.orchestra.run.smithi139.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 27 00:11 /home/ubuntu/cephtest 2024-07-27T00:11:29.050 INFO:teuthology.orchestra.run.smithi177.stdout: 265723 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 27 00:11 /home/ubuntu/cephtest 2024-07-27T00:11:29.051 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-07-27T00:11:29.064 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-07-27T00:11:29.099 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-07-27T00:11:29.102 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3188.9668390750885 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-07-27T00:11:29.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-27T00:11:29.179 INFO:teuthology.run:FAIL